[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:20:54 dist-test-slave-rmhg krb5kdc[22543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:20:54 dist-test-slave-rmhg krb5kdc[22543](info): set up 2 sockets
Aug 09 05:20:54 dist-test-slave-rmhg krb5kdc[22543](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260809 05:20:55.245867 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:20:55 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:20:55.435204 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:45981
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:45981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:20:55.814733 22559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:20:55.815167 22559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:20:55.815310 22559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:20:55.822572 22559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:20:55.822717 22559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:20:55.822799 22559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:20:55.822875 22559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:20:55.822939 22559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:20:55.833510 22559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:45981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:20:55.835624 22559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:20:55.837615 22559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:20:55.856173 22564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:55.856207 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:55.856237 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:20:55.858400 22559 server_base.cc:1034] running on GCE node
I20260809 05:20:55.859581 22559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:20:55.863149 22559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:20:55.864450 22559 hybrid_clock.cc:648] HybridClock initialized: now 1786252855864346 us; error 124 us; skew 500 ppm
I20260809 05:20:55.865002 22559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:20:55 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252855, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:20:55.872221 22559 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:20:55.874971 22559 webserver.cc:460] Webserver started at http://127.22.1.126:37569/ using document root <none> and password file <none>
I20260809 05:20:55.876099 22559 fs_manager.cc:359] Metadata directory not provided
I20260809 05:20:55.876221 22559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:20:55.876734 22559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:20:55.880962 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "106b059fab3b475a8d5194dda8aa9885"
format_stamp: "Formatted at 2026-08-09 05:20:55 on dist-test-slave-rmhg"
server_key: "78208188d13af0bd0ab4935a428f3d55"
server_key_iv: "ef7a825bacdba4105bd90251cd18f710"
server_key_version: "encryptionkey@0"
I20260809 05:20:55.881771 22559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "106b059fab3b475a8d5194dda8aa9885"
format_stamp: "Formatted at 2026-08-09 05:20:55 on dist-test-slave-rmhg"
server_key: "78208188d13af0bd0ab4935a428f3d55"
server_key_iv: "ef7a825bacdba4105bd90251cd18f710"
server_key_version: "encryptionkey@0"
I20260809 05:20:55.891006 22559 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260809 05:20:55.896777 22582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:55.898788 22559 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260809 05:20:55.899060 22559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "106b059fab3b475a8d5194dda8aa9885"
format_stamp: "Formatted at 2026-08-09 05:20:55 on dist-test-slave-rmhg"
server_key: "78208188d13af0bd0ab4935a428f3d55"
server_key_iv: "ef7a825bacdba4105bd90251cd18f710"
server_key_version: "encryptionkey@0"
I20260809 05:20:55.899243 22559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:20:55.923283 22559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:20:55.928614 22559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:20:55.928946 22559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:20:55.953068 22559 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:45981
I20260809 05:20:55.953118 22655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:45981 every 8 connection(s)
I20260809 05:20:55.955582 22559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:20:55.962939 22657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:20:55.966480 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22559
I20260809 05:20:55.966928 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:20:55.971812 22533 external_mini_cluster.cc:1428] Setting key 520aaba2fb10da97209eb97068a5177f
I20260809 05:20:55.983199 22657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885: Bootstrap starting.
I20260809 05:20:55.990876 22657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885: Neither blocks nor log segments found. Creating new log.
I20260809 05:20:55.993065 22657 log.cc:824] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885: Log is configured to *not* fsync() on all Append() calls
I20260809 05:20:55.998413 22657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885: No bootstrap required, opened a new log
I20260809 05:20:56.010622 22657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } }
I20260809 05:20:56.011286 22657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:20:56.011510 22657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 106b059fab3b475a8d5194dda8aa9885, State: Initialized, Role: FOLLOWER
I20260809 05:20:56.012949 22657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } }
Aug 09 05:20:56 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:20:56.013360 22657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:20:56.013669 22657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:20:56.013890 22657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:20:56.017396 22657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } }
I20260809 05:20:56.018731 22657 leader_election.cc:302] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 106b059fab3b475a8d5194dda8aa9885; no voters:
I20260809 05:20:56.019699 22657 leader_election.cc:288] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:20:56.019996 22663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:20:56.021235 22663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [term 1 LEADER]: Becoming Leader. State: Replica: 106b059fab3b475a8d5194dda8aa9885, State: Running, Role: LEADER
I20260809 05:20:56.022131 22663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } }
I20260809 05:20:56.023159 22657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:20:56.027375 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:55.982568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.22.1.126:45981)
0809 05:20:55.983624 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:55.983636 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:55.983682 (+ 46us) server_negotiation.cc:407] Connection header received
0809 05:20:55.996276 (+ 12594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:55.996300 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:55.996930 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:55.997367 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:56.002258 (+ 4891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.003900 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.006551 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.007265 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.007556 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:56.014857 (+ 7301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:56.014954 (+ 97us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:56.015006 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:56.015078 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:56.019775 (+ 4697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:56.021022 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:56.021030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:56.021038 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:56.021372 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:56.022020 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:56.022026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:56.022030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:56.023018 (+ 988us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:56.023450 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:56.024725 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0809 05:20:56.025241 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":416,"threads_started":1}
I20260809 05:20:56.028735 22665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 106b059fab3b475a8d5194dda8aa9885. Latest consensus state: current_term: 1 leader_uuid: "106b059fab3b475a8d5194dda8aa9885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } } }
I20260809 05:20:56.029145 22665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [sys.catalog]: This master's current role is: LEADER
I20260809 05:20:56.029747 22664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "106b059fab3b475a8d5194dda8aa9885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "106b059fab3b475a8d5194dda8aa9885" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45981 } } }
I20260809 05:20:56.031992 22664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885 [sys.catalog]: This master's current role is: LEADER
I20260809 05:20:56.039484 22671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:20:56.047828 22671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:20:56.062359 22671 catalog_manager.cc:1347] Generated new cluster ID: 8f104876517b4d06bdef0c17a6163265
I20260809 05:20:56.062492 22671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:20:56.084574 22671 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:20:56.086488 22671 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:20:56.096706 22671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 106b059fab3b475a8d5194dda8aa9885: Generated new TSK 0
I20260809 05:20:56.097839 22671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:20:56.165089 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:20:56.520762 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:20:56.521046 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:20:56.521183 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:20:56.529136 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:20:56.529284 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:20:56.529449 22694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:20:56.539951 22694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:20:56.541301 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:20:56.542649 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:20:56.556607 22702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:56.556573 22705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:56.556638 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:20:56.557411 22694 server_base.cc:1034] running on GCE node
I20260809 05:20:56.558022 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:20:56.558997 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:20:56.560223 22694 hybrid_clock.cc:648] HybridClock initialized: now 1786252856560170 us; error 46 us; skew 500 ppm
I20260809 05:20:56.560570 22694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:20:56 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252856, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:20:56.563712 22694 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:20:56.565299 22694 webserver.cc:460] Webserver started at http://127.22.1.65:33553/ using document root <none> and password file <none>
I20260809 05:20:56.565999 22694 fs_manager.cc:359] Metadata directory not provided
I20260809 05:20:56.566123 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:20:56.566434 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:20:56.569576 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "051d76fb814b4bac820c2e12e89d6288"
format_stamp: "Formatted at 2026-08-09 05:20:56 on dist-test-slave-rmhg"
server_key: "9874e39f947e9ce83b1db4f7a5523685"
server_key_iv: "3eeeca06f1953414a6eb865fe2c3bd44"
server_key_version: "encryptionkey@0"
I20260809 05:20:56.570324 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "051d76fb814b4bac820c2e12e89d6288"
format_stamp: "Formatted at 2026-08-09 05:20:56 on dist-test-slave-rmhg"
server_key: "9874e39f947e9ce83b1db4f7a5523685"
server_key_iv: "3eeeca06f1953414a6eb865fe2c3bd44"
server_key_version: "encryptionkey@0"
I20260809 05:20:56.576081 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:20:56.579813 22718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:56.581322 22694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:20:56.581501 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "051d76fb814b4bac820c2e12e89d6288"
format_stamp: "Formatted at 2026-08-09 05:20:56 on dist-test-slave-rmhg"
server_key: "9874e39f947e9ce83b1db4f7a5523685"
server_key_iv: "3eeeca06f1953414a6eb865fe2c3bd44"
server_key_version: "encryptionkey@0"
I20260809 05:20:56.581707 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:20:56.606065 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:20:56.610497 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:20:56.610773 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:20:56.611956 22694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:20:56.613708 22694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:20:56.613796 22694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:56.613986 22694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:20:56.614053 22694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:56.646337 22694 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:36901
I20260809 05:20:56.646400 22883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:36901 every 8 connection(s)
I20260809 05:20:56.647714 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:20:56.650134 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22694
I20260809 05:20:56.650357 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:20:56.651129 22533 external_mini_cluster.cc:1428] Setting key b25ec9b5be54b6c211379edd8f781caf
Aug 09 05:20:56 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252856, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
I20260809 05:20:56.671144 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:56.651444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:54193 (local address 127.22.1.126:45981)
0809 05:20:56.651881 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:56.651898 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:56.655240 (+ 3342us) server_negotiation.cc:407] Connection header received
0809 05:20:56.655297 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:56.655300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:56.655364 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:56.655443 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:56.657589 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.658340 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.660086 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.660530 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.660633 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:56.664498 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:56.664507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:56.664524 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:56.664568 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:56.667748 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:56.668585 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:56.668591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:56.668596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:56.668770 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:56.669247 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:56.669252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:56.669256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:56.669638 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:56.669792 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:56.670690 (+ 898us) server_negotiation.cc:299] Negotiation successful
0809 05:20:56.670881 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":222,"threads_started":1}
I20260809 05:20:56.672219 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:56.652176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45981 (local address 127.22.1.65:54193)
0809 05:20:56.653098 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0809 05:20:56.653184 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0809 05:20:56.654932 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:20:56.655807 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:20:56.655819 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:20:56.656559 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:20:56.657384 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:56.657411 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.658516 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.658521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:56.659932 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:56.659946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:56.660754 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:56.660760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:56.660974 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:56.661766 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:20:56.661791 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:20:56.664212 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:20:56.667920 (+ 3708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:56.667928 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:56.667946 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:56.668417 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:56.668890 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:56.668894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:56.668896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:56.669098 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:56.669790 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:20:56.669797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:20:56.670173 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 05:20:56.670547 (+ 374us) client_negotiation.cc:240] Negotiation successful
0809 05:20:56.671023 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":340,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
I20260809 05:20:56.674155 22886 heartbeater.cc:347] Connected to a master server at 127.22.1.126:45981
I20260809 05:20:56.674615 22886 heartbeater.cc:464] Registering TS with master...
I20260809 05:20:56.675500 22886 heartbeater.cc:511] Master 127.22.1.126:45981 requested a full tablet report, sending...
I20260809 05:20:56.679644 22610 ts_manager.cc:194] Registered new tserver with Master: 051d76fb814b4bac820c2e12e89d6288 (127.22.1.65:36901)
I20260809 05:20:56.681864 22610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:54193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:20:56.694716 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:20:57.026022 22899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:20:57.026341 22899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:20:57.026486 22899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:20:57.034264 22899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:20:57.034397 22899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:20:57.034556 22899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:20:57.044515 22899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:20:57.045904 22899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:20:57.047271 22899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:20:57.060312 22909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:57.060398 22911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:57.060884 22907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:20:57.061077 22899 server_base.cc:1034] running on GCE node
I20260809 05:20:57.061766 22899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:20:57.062657 22899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:20:57.063865 22899 hybrid_clock.cc:648] HybridClock initialized: now 1786252857063816 us; error 38 us; skew 500 ppm
I20260809 05:20:57.064180 22899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:20:57 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252857, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:20:57.067232 22899 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:20:57.068735 22899 webserver.cc:460] Webserver started at http://127.22.1.66:42679/ using document root <none> and password file <none>
I20260809 05:20:57.069453 22899 fs_manager.cc:359] Metadata directory not provided
I20260809 05:20:57.069563 22899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:20:57.069828 22899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:20:57.072690 22899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "681f7a6182ac4df2a079bd754b34ab2e"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "5b1bfcaf8de882a246b97c4baac4e57a"
server_key_iv: "65cc58f685dc1f866f3a074e509c747b"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.073475 22899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "681f7a6182ac4df2a079bd754b34ab2e"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "5b1bfcaf8de882a246b97c4baac4e57a"
server_key_iv: "65cc58f685dc1f866f3a074e509c747b"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.078662 22899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 05:20:57.083320 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:57.084879 22899 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260809 05:20:57.085065 22899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "681f7a6182ac4df2a079bd754b34ab2e"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "5b1bfcaf8de882a246b97c4baac4e57a"
server_key_iv: "65cc58f685dc1f866f3a074e509c747b"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.085351 22899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:20:57.114287 22899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:20:57.118513 22899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:20:57.118789 22899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:20:57.119776 22899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:20:57.121109 22899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:20:57.121220 22899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 05:20:57.121341 22899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:20:57.121395 22899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:57.186170 22899 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:37629
I20260809 05:20:57.186270 23087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:37629 every 8 connection(s)
I20260809 05:20:57.187858 22899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:20:57.194056 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22899
I20260809 05:20:57.194274 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:20:57.194902 22533 external_mini_cluster.cc:1428] Setting key 7131d685a7c2a8886c93566180eecf50
Aug 09 05:20:57 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252857, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:20:57.216593 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:57.191524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:42957 (local address 127.22.1.126:45981)
0809 05:20:57.192285 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:57.192290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:57.194127 (+ 1837us) server_negotiation.cc:407] Connection header received
0809 05:20:57.197251 (+ 3124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:57.197257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:57.197340 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:57.197597 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:57.199552 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:57.201015 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:57.203176 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:57.204177 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:57.204317 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:57.209378 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:57.209383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:57.209386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:57.209420 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:57.213134 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:57.214000 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:57.214004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:57.214007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:57.214193 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:57.214712 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:57.214717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:57.214720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:57.215039 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:57.215173 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:57.215969 (+ 796us) server_negotiation.cc:299] Negotiation successful
0809 05:20:57.216183 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":516,"threads_started":1}
I20260809 05:20:57.217793 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:57.192207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45981 (local address 127.22.1.66:42957)
0809 05:20:57.193758 (+ 1551us) negotiation.cc:107] Waiting for socket to connect
0809 05:20:57.193822 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 05:20:57.196882 (+ 3060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:20:57.197816 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:20:57.197826 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:20:57.198505 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:20:57.199353 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:57.199375 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:57.201210 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:57.201215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:57.203031 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:57.203045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:57.204560 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:57.204569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:57.204908 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:57.205723 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:20:57.205747 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:20:57.209114 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:20:57.213285 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:57.213295 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:57.213312 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:57.213866 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:57.214305 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:57.214310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:57.214314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:57.214588 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:57.215167 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:20:57.215175 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:20:57.215551 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 05:20:57.215840 (+ 289us) client_negotiation.cc:240] Negotiation successful
0809 05:20:57.216379 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":875,"threads_started":1}
I20260809 05:20:57.219560 23088 heartbeater.cc:347] Connected to a master server at 127.22.1.126:45981
I20260809 05:20:57.219938 23088 heartbeater.cc:464] Registering TS with master...
I20260809 05:20:57.220978 23088 heartbeater.cc:511] Master 127.22.1.126:45981 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:20:57.222718 22610 ts_manager.cc:194] Registered new tserver with Master: 681f7a6182ac4df2a079bd754b34ab2e (127.22.1.66:37629)
I20260809 05:20:57.223829 22610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:42957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:20:57.250145 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:20:57.685768 22886 heartbeater.cc:503] Master 127.22.1.126:45981 was elected leader, sending a full tablet report...
W20260809 05:20:57.823587 23095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:20:57.824018 23095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:20:57.824211 23095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:20:57.833868 23095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:20:57.834039 23095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:20:57.834247 23095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:20:57.846448 23095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:45981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:20:57.847893 23095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:20:57.849411 23095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:20:57.866171 23103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:57.867273 23104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:20:57.867304 23106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:20:57.868093 23095 server_base.cc:1034] running on GCE node
I20260809 05:20:57.868690 23095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:20:57.869743 23095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:20:57.870946 23095 hybrid_clock.cc:648] HybridClock initialized: now 1786252857870884 us; error 50 us; skew 500 ppm
I20260809 05:20:57.871269 23095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:20:57 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252857, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:20:57.875895 23095 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:20:57.877714 23095 webserver.cc:460] Webserver started at http://127.22.1.67:41205/ using document root <none> and password file <none>
I20260809 05:20:57.878551 23095 fs_manager.cc:359] Metadata directory not provided
I20260809 05:20:57.878700 23095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:20:57.879089 23095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:20:57.882522 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "7101594df6e340d2a7a19052743e14e5"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "bc93799246b37d8b9313b3d774ad7450"
server_key_iv: "23249c1ecfb6a0cc4ce3fded6ac95ce3"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.883478 23095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "7101594df6e340d2a7a19052743e14e5"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "bc93799246b37d8b9313b3d774ad7450"
server_key_iv: "23249c1ecfb6a0cc4ce3fded6ac95ce3"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.890877 23095 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260809 05:20:57.895572 23118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:57.897217 23095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:20:57.897466 23095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "7101594df6e340d2a7a19052743e14e5"
format_stamp: "Formatted at 2026-08-09 05:20:57 on dist-test-slave-rmhg"
server_key: "bc93799246b37d8b9313b3d774ad7450"
server_key_iv: "23249c1ecfb6a0cc4ce3fded6ac95ce3"
server_key_version: "encryptionkey@0"
I20260809 05:20:57.897737 23095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:20:57.930611 23095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:20:57.935693 23095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:20:57.936280 23095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:20:57.937528 23095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:20:57.939311 23095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:20:57.939427 23095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:57.939664 23095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:20:57.939790 23095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:20:58.019906 23095 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40619
I20260809 05:20:58.020025 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40619 every 8 connection(s)
I20260809 05:20:58.021633 23095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:20:58.026301 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23095
I20260809 05:20:58.026531 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:20:58.027141 22533 external_mini_cluster.cc:1428] Setting key 96b953b86c9957a1b93999fd5e875e7a
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252857, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:20:58.047806 23277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.024913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:54299 (local address 127.22.1.126:45981)
0809 05:20:58.025764 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.025769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.027550 (+ 1781us) server_negotiation.cc:407] Connection header received
0809 05:20:58.030511 (+ 2961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.030516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.030606 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.030839 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.033177 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.034048 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.036066 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.036467 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.036575 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.040586 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.040591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.040594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.040628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.044196 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.045046 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.045051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.045053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.045297 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.045770 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.045776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.045788 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.046160 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.046343 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.047084 (+ 741us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.047349 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":591,"threads_started":1}
I20260809 05:20:58.048887 23278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.025869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45981 (local address 127.22.1.67:54299)
0809 05:20:58.027224 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0809 05:20:58.027305 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0809 05:20:58.030260 (+ 2955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:20:58.031176 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:20:58.031190 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:20:58.032094 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:20:58.032993 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:58.033014 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.034226 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.034230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:58.035925 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:20:58.035937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.036727 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.036735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:20:58.036974 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.037669 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:20:58.037692 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:20:58.040361 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:20:58.044384 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:58.044393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:58.044408 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:58.044919 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:58.045410 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:20:58.045414 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:20:58.045416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:20:58.045657 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:20:58.046302 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:20:58.046311 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:20:58.046667 (+ 356us) client_negotiation.cc:769] Sending connection context
0809 05:20:58.046941 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 05:20:58.047490 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":798,"threads_started":1}
I20260809 05:20:58.050419 23274 heartbeater.cc:347] Connected to a master server at 127.22.1.126:45981
I20260809 05:20:58.050752 23274 heartbeater.cc:464] Registering TS with master...
I20260809 05:20:58.051494 23274 heartbeater.cc:511] Master 127.22.1.126:45981 requested a full tablet report, sending...
I20260809 05:20:58.053022 22610 ts_manager.cc:194] Registered new tserver with Master: 7101594df6e340d2a7a19052743e14e5 (127.22.1.67:40619)
I20260809 05:20:58.054095 22610 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:54299
I20260809 05:20:58.056775 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:20:58.122704 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.102675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.22.1.65:36901)
0809 05:20:58.103561 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.103571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.104033 (+ 462us) server_negotiation.cc:407] Connection header received
0809 05:20:58.104328 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.104336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.104616 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.104770 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.106726 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.107783 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.109497 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.110109 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.110210 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.116016 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.116032 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.116050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.116090 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.119628 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.120314 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.120321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.120324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.120536 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.120839 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.120842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.120844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.121355 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.121760 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.121933 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.122173 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":436,"threads_started":1}
W20260809 05:20:58.124413 22756 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53608
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:20:58.180166 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.162553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.22.1.65:36901)
0809 05:20:58.162758 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.162766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.163378 (+ 612us) server_negotiation.cc:407] Connection header received
0809 05:20:58.163554 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.163558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.163657 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.163745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.165264 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.166044 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.167414 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.167918 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.168030 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.173657 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.173662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.173665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.173702 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.177187 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.178003 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.178009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.178012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.178228 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.178660 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.178664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.178666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.178973 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.179196 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.179561 (+ 365us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.179791 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 05:20:58.182403 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22694
I20260809 05:20:58.200050 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.183473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.22.1.65:36901)
0809 05:20:58.183765 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.183771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.183966 (+ 195us) server_negotiation.cc:407] Connection header received
0809 05:20:58.184088 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.184093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.184166 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.184286 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.185841 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.187309 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.189025 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.190191 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.190360 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.193556 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.193562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.193566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.193596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.197020 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.197841 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.197845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.197847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.198076 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.198517 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.198521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.198524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.198797 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.199057 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.199408 (+ 351us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.199577 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 05:20:58.200954 22872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:20:58.227051 23088 heartbeater.cc:503] Master 127.22.1.126:45981 was elected leader, sending a full tablet report...
I20260809 05:20:58.291740 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22694
I20260809 05:20:58.313827 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22899
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:20:58.330616 23293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.314873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.22.1.66:37629)
0809 05:20:58.315542 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.315550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.315575 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:20:58.315635 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.315641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.315851 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.315980 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.317113 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.318178 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.319404 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.320021 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.320140 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.324635 (+ 4495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.324650 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.324667 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.324711 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.327952 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.328631 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.328639 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.328641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.328788 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.329093 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.329096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.329098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.329525 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.329760 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.330022 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.330242 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":342,"threads_started":1}
I20260809 05:20:58.332036 23078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:20:58.416240 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22899
I20260809 05:20:58.434530 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23095
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[22543](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:20:58.450280 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:20:58.435588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.22.1.67:40619)
0809 05:20:58.436102 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 05:20:58.436109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:20:58.436139 (+ 30us) server_negotiation.cc:407] Connection header received
0809 05:20:58.436199 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:20:58.436205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:20:58.436414 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:20:58.436587 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:20:58.437851 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.438865 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.440203 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:20:58.440744 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:20:58.440858 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:20:58.444794 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:20:58.444805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:20:58.444821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:20:58.444853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:20:58.447567 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.448204 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.448212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.448214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.448338 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:20:58.448698 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:20:58.448702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:20:58.448704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:20:58.449062 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:20:58.449382 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0809 05:20:58.449682 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 05:20:58.449939 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":246,"threads_started":1}
I20260809 05:20:58.451593 23266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:20:58.533674 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23095
I20260809 05:20:58.557405 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22559
I20260809 05:20:58.558729 22650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:20:58.660633 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22559
2026-08-09T05:20:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4630 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[23304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[23304](info): set up 2 sockets
Aug 09 05:20:58 dist-test-slave-rmhg krb5kdc[23304](info): commencing operation
krb5kdc: starting...
W20260809 05:20:59.917743 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260809 05:21:00.241449 22533 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.323s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260809 05:21:00.411806 22533 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.170s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260809 05:21:00.516662 22533 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.105s user 0.001s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/test-user.keytab.
W20260809 05:21:01.067462 22533 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.551s user 0.003s sys 0.003s
Aug 09 05:21:01 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
W20260809 05:21:01.292299 22533 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.22.1.126: real 0.172s user 0.003s sys 0.007s
I20260809 05:21:01.293433 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:34885
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:01.623061 23322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:01.623394 23322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:01.623535 23322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:01.630929 23322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:01.631072 23322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:01.631155 23322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:01.631232 23322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:01.631289 23322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:01.641481 23322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:01.642926 23322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:01.644305 23322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:01.656708 23330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:01.656764 23329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:01.656744 23333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:01.657713 23322 server_base.cc:1034] running on GCE node
I20260809 05:21:01.658594 23322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:01.660115 23322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:01.661334 23322 hybrid_clock.cc:648] HybridClock initialized: now 1786252861661262 us; error 60 us; skew 500 ppm
I20260809 05:21:01.661733 23322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:01 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252861, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:01.668469 23322 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:01.670212 23322 webserver.cc:460] Webserver started at http://127.22.1.126:39051/ using document root <none> and password file <none>
I20260809 05:21:01.670985 23322 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:01.671101 23322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:01.671379 23322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:01.674499 23322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "002e80ad50af4f0c800e692360806a71"
format_stamp: "Formatted at 2026-08-09 05:21:01 on dist-test-slave-rmhg"
server_key: "5cc0bb2e1978fabf2c5fd75c5e5ebbf7"
server_key_iv: "60e9954f4f2066fcb6a72e4bbe0cd702"
server_key_version: "encryptionkey@0"
I20260809 05:21:01.675225 23322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "002e80ad50af4f0c800e692360806a71"
format_stamp: "Formatted at 2026-08-09 05:21:01 on dist-test-slave-rmhg"
server_key: "5cc0bb2e1978fabf2c5fd75c5e5ebbf7"
server_key_iv: "60e9954f4f2066fcb6a72e4bbe0cd702"
server_key_version: "encryptionkey@0"
I20260809 05:21:01.680655 23322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260809 05:21:01.684350 23342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:01.685920 23322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:01.686074 23322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "002e80ad50af4f0c800e692360806a71"
format_stamp: "Formatted at 2026-08-09 05:21:01 on dist-test-slave-rmhg"
server_key: "5cc0bb2e1978fabf2c5fd75c5e5ebbf7"
server_key_iv: "60e9954f4f2066fcb6a72e4bbe0cd702"
server_key_version: "encryptionkey@0"
I20260809 05:21:01.686252 23322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:01.718212 23322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:01.722806 23322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:01.723070 23322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:01.741086 23322 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:34885
I20260809 05:21:01.741130 23415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:34885 every 8 connection(s)
I20260809 05:21:01.742558 23322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:01.743165 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23322
I20260809 05:21:01.743359 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:01.744135 22533 external_mini_cluster.cc:1428] Setting key 76ea91043352d0950675fd76747491dd
I20260809 05:21:01.746984 23416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 05:21:01 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:01.759897 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: Bootstrap starting.
I20260809 05:21:01.764058 23416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:01.765327 23416 log.cc:824] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:01.767853 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:01.746184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.22.1.126:34885)
0809 05:21:01.746938 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:01.746950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:01.746996 (+ 46us) server_negotiation.cc:407] Connection header received
0809 05:21:01.748657 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:01.748682 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:01.749052 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:01.749439 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:01.751225 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:01.752571 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:01.754373 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:01.755035 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:01.755182 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:01.759763 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:01.759775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:01.759795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:01.759842 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:01.763574 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:01.764301 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:01.764309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:01.764316 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:01.764514 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:01.764918 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:01.764921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:01.764923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:01.765534 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:01.765763 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:01.766413 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 05:21:01.766831 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":328,"threads_started":1}
I20260809 05:21:01.768373 23416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: No bootstrap required, opened a new log
I20260809 05:21:01.776774 23416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } }
I20260809 05:21:01.777191 23416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:01.777292 23416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 002e80ad50af4f0c800e692360806a71, State: Initialized, Role: FOLLOWER
I20260809 05:21:01.777885 23416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } }
I20260809 05:21:01.778110 23416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:01.778240 23416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:01.778403 23416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:01.781020 23416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } }
I20260809 05:21:01.781626 23416 leader_election.cc:302] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 002e80ad50af4f0c800e692360806a71; no voters:
I20260809 05:21:01.782177 23416 leader_election.cc:288] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:01.782307 23424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:01.784010 23416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:01.789028 23424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [term 1 LEADER]: Becoming Leader. State: Replica: 002e80ad50af4f0c800e692360806a71, State: Running, Role: LEADER
I20260809 05:21:01.789541 23424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } }
I20260809 05:21:01.792732 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 002e80ad50af4f0c800e692360806a71. Latest consensus state: current_term: 1 leader_uuid: "002e80ad50af4f0c800e692360806a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } } }
I20260809 05:21:01.793030 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "002e80ad50af4f0c800e692360806a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "002e80ad50af4f0c800e692360806a71" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34885 } } }
I20260809 05:21:01.793270 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:01.793342 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71 [sys.catalog]: This master's current role is: LEADER
W20260809 05:21:01.797966 23441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:21:01.798094 23441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:21:01.798182 23443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:01.799795 23443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:01.808455 23443 catalog_manager.cc:1347] Generated new cluster ID: c0990b3464e94ed8a37976adfa31a560
I20260809 05:21:01.808581 23443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:01.817667 23443 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:01.819514 23443 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:01.826455 23443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 002e80ad50af4f0c800e692360806a71: Generated new TSK 0
I20260809 05:21:01.827510 23443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:01.876228 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:02.204567 23451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:02.204864 23451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:02.204993 23451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:02.212333 23451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:02.212457 23451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:02.212617 23451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:02.222504 23451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:02.223863 23451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:02.225272 23451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:02.238034 23457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:02.238063 23458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:02.238260 23461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:02.238909 23451 server_base.cc:1034] running on GCE node
I20260809 05:21:02.239562 23451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:02.240561 23451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:02.241781 23451 hybrid_clock.cc:648] HybridClock initialized: now 1786252862241723 us; error 45 us; skew 500 ppm
I20260809 05:21:02.242113 23451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:02 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252862, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:02.245409 23451 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:02.246929 23451 webserver.cc:460] Webserver started at http://127.22.1.65:32923/ using document root <none> and password file <none>
I20260809 05:21:02.247659 23451 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:02.247776 23451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:02.248100 23451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:02.251199 23451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "4a019f3c520e4051902d86411fdd1ee5"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "281dc4a40aa17c5fa38c41d9d170df9a"
server_key_iv: "941a65efaf195cf7605de21e55f40899"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.252089 23451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "4a019f3c520e4051902d86411fdd1ee5"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "281dc4a40aa17c5fa38c41d9d170df9a"
server_key_iv: "941a65efaf195cf7605de21e55f40899"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.257339 23451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:02.260716 23469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.262153 23451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:02.262303 23451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "4a019f3c520e4051902d86411fdd1ee5"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "281dc4a40aa17c5fa38c41d9d170df9a"
server_key_iv: "941a65efaf195cf7605de21e55f40899"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.262476 23451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:02.277333 23451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:02.281343 23451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:02.281617 23451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:02.282594 23451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:02.284024 23451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:02.284144 23451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.284255 23451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:02.284305 23451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.314800 23451 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:39393
I20260809 05:21:02.314848 23630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:39393 every 8 connection(s)
I20260809 05:21:02.316082 23451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:02.316588 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23451
I20260809 05:21:02.316766 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:02.317623 22533 external_mini_cluster.cc:1428] Setting key 0237ee8e208b567589a66bf3fb5af5b0
Aug 09 05:21:02 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252862, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
I20260809 05:21:02.337930 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:02.319344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:43941 (local address 127.22.1.126:34885)
0809 05:21:02.319809 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:02.319816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:02.320293 (+ 477us) server_negotiation.cc:407] Connection header received
0809 05:21:02.322214 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:02.322220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:02.322309 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:02.322403 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:02.324095 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.324829 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.327069 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.327651 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.327788 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:02.331495 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:02.331502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:02.331507 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:02.331551 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:02.334846 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:02.335619 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:02.335624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:02.335628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:02.335779 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:02.336196 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:02.336201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:02.336205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:02.336582 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:02.336734 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:02.337452 (+ 718us) server_negotiation.cc:299] Negotiation successful
0809 05:21:02.337650 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260809 05:21:02.338974 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:02.319310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34885 (local address 127.22.1.65:43941)
0809 05:21:02.320144 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:02.320185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:02.321926 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:02.322709 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:02.322720 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:02.323277 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:02.323912 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:02.323931 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.325004 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.325009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:02.326907 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:02.326921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.327878 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.327884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:02.328100 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:02.328765 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:02.328789 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:02.331181 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:02.335014 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:02.335023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:02.335038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:02.335466 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:02.335882 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:02.335885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:02.335887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:02.336076 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:02.336689 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:02.336695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:02.337075 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 05:21:02.337347 (+ 272us) client_negotiation.cc:240] Negotiation successful
0809 05:21:02.337825 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":349,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
I20260809 05:21:02.340488 23631 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34885
I20260809 05:21:02.340803 23631 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:02.341599 23631 heartbeater.cc:511] Master 127.22.1.126:34885 requested a full tablet report, sending...
I20260809 05:21:02.343593 23366 ts_manager.cc:194] Registered new tserver with Master: 4a019f3c520e4051902d86411fdd1ee5 (127.22.1.65:39393)
I20260809 05:21:02.345508 23366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:43941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:02.361765 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:02.688591 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:02.688874 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:02.688968 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:02.696226 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:02.696360 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:02.696480 23637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:02.706374 23637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:02.707806 23637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:02.709429 23637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:02.722613 23650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:02.722669 23651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:02.722673 23653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:02.723641 23637 server_base.cc:1034] running on GCE node
I20260809 05:21:02.724265 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:02.725323 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:02.726528 23637 hybrid_clock.cc:648] HybridClock initialized: now 1786252862726477 us; error 46 us; skew 500 ppm
I20260809 05:21:02.726867 23637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:02 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252862, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:02.730163 23637 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:02.731678 23637 webserver.cc:460] Webserver started at http://127.22.1.66:46547/ using document root <none> and password file <none>
I20260809 05:21:02.732355 23637 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:02.732481 23637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:02.732786 23637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:02.735754 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "bd3903a2fc1a45d383958e75169917c4"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "aa7328420c0e62c4c20c4a14324ef64a"
server_key_iv: "ede8e7ac79bf157f3f4f500964060752"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.736541 23637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "bd3903a2fc1a45d383958e75169917c4"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "aa7328420c0e62c4c20c4a14324ef64a"
server_key_iv: "ede8e7ac79bf157f3f4f500964060752"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.741915 23637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 05:21:02.745538 23664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.746937 23637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:02.747098 23637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "bd3903a2fc1a45d383958e75169917c4"
format_stamp: "Formatted at 2026-08-09 05:21:02 on dist-test-slave-rmhg"
server_key: "aa7328420c0e62c4c20c4a14324ef64a"
server_key_iv: "ede8e7ac79bf157f3f4f500964060752"
server_key_version: "encryptionkey@0"
I20260809 05:21:02.747283 23637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:02.764369 23637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:02.768786 23637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:02.769070 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:02.770191 23637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:02.771588 23637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:02.771689 23637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.771817 23637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:02.771876 23637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:02.803362 23637 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:40061
I20260809 05:21:02.803409 23827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:40061 every 8 connection(s)
I20260809 05:21:02.804669 23637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:02.811079 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23637
I20260809 05:21:02.811260 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:02.811822 22533 external_mini_cluster.cc:1428] Setting key 80590268262448eee826603e1864dc60
Aug 09 05:21:02 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252862, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:02.827263 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:02.807145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:34023 (local address 127.22.1.126:34885)
0809 05:21:02.807361 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:02.807367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:02.808626 (+ 1259us) server_negotiation.cc:407] Connection header received
0809 05:21:02.810472 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:02.810476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:02.810556 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:02.810644 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:02.812656 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.813666 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.815881 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.816485 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.816578 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:02.820482 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:02.820490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:02.820494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:02.820533 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:02.823725 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:02.824616 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:02.824622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:02.824626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:02.824851 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:02.825409 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:02.825414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:02.825418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:02.825726 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:02.825867 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:02.826645 (+ 778us) server_negotiation.cc:299] Negotiation successful
0809 05:21:02.826920 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:21:02.828341 23830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:02.807740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34885 (local address 127.22.1.66:34023)
0809 05:21:02.808458 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:02.808504 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:02.810178 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:02.810928 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:02.810940 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:02.811681 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:02.812476 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:02.812497 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.813815 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.813819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:02.815706 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:02.815723 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:02.816925 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:02.816932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:02.817260 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:02.817960 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:02.817988 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:02.820249 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:02.823890 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:02.823898 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:02.823914 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:02.824398 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:02.824986 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:02.824991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:02.824995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:02.825303 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:02.825857 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:02.825865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:02.826245 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 05:21:02.826542 (+ 297us) client_negotiation.cc:240] Negotiation successful
0809 05:21:02.827134 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":293,"threads_started":1}
I20260809 05:21:02.829870 23828 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34885
I20260809 05:21:02.830227 23828 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:02.831200 23828 heartbeater.cc:511] Master 127.22.1.126:34885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:21:02.832899 23366 ts_manager.cc:194] Registered new tserver with Master: bd3903a2fc1a45d383958e75169917c4 (127.22.1.66:40061)
I20260809 05:21:02.833724 23366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:34023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:02.855515 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:03.180145 23834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:03.180433 23834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:03.180564 23834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:03.187958 23834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:03.188072 23834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:03.188230 23834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:03.198182 23834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34885
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:03.199513 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:03.200903 23834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:03.214519 23842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:03.214478 23843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:03.214452 23846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:03.214921 23834 server_base.cc:1034] running on GCE node
I20260809 05:21:03.215790 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:03.216712 23834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:03.217916 23834 hybrid_clock.cc:648] HybridClock initialized: now 1786252863217857 us; error 42 us; skew 500 ppm
I20260809 05:21:03.218223 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:03.221485 23834 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:03.222956 23834 webserver.cc:460] Webserver started at http://127.22.1.67:41753/ using document root <none> and password file <none>
I20260809 05:21:03.223631 23834 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:03.223747 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:03.224057 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:03.227063 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "fa8b207bd3944daa877a42823c626a65"
format_stamp: "Formatted at 2026-08-09 05:21:03 on dist-test-slave-rmhg"
server_key: "0688d470ae3b1e4896064765b0abb1aa"
server_key_iv: "049bee404846f084bb38ac9349c4b712"
server_key_version: "encryptionkey@0"
I20260809 05:21:03.227928 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "fa8b207bd3944daa877a42823c626a65"
format_stamp: "Formatted at 2026-08-09 05:21:03 on dist-test-slave-rmhg"
server_key: "0688d470ae3b1e4896064765b0abb1aa"
server_key_iv: "049bee404846f084bb38ac9349c4b712"
server_key_version: "encryptionkey@0"
I20260809 05:21:03.233021 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:03.236575 23861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:03.237900 23834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:03.238059 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "fa8b207bd3944daa877a42823c626a65"
format_stamp: "Formatted at 2026-08-09 05:21:03 on dist-test-slave-rmhg"
server_key: "0688d470ae3b1e4896064765b0abb1aa"
server_key_iv: "049bee404846f084bb38ac9349c4b712"
server_key_version: "encryptionkey@0"
I20260809 05:21:03.238237 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:03.250912 23834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:03.255084 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:03.255340 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:03.256268 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:03.257587 23834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:03.257665 23834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:03.257777 23834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:03.257835 23834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:03.300160 23834 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:45927
I20260809 05:21:03.300257 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:45927 every 8 connection(s)
I20260809 05:21:03.301761 23834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:03.304085 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23834
I20260809 05:21:03.304354 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:03.305066 22533 external_mini_cluster.cc:1428] Setting key 2ca2fe5a84113462bc2c6d4f9a819b80
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:03.330176 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.305552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:39757 (local address 127.22.1.126:34885)
0809 05:21:03.305791 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.305797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.308033 (+ 2236us) server_negotiation.cc:407] Connection header received
0809 05:21:03.310614 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.310620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.310772 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.310866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.312927 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.314049 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.316323 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.317210 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.317360 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.322834 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.322840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.322844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.322894 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.326170 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.327181 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.327185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.327188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.327323 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.327987 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.327990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.327992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.328376 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.328499 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.329377 (+ 878us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.329637 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:21:03.331506 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.306319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34885 (local address 127.22.1.67:39757)
0809 05:21:03.307875 (+ 1556us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.307925 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.310314 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.311164 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.311174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.311920 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:03.312740 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.312762 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.314286 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.314292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.316158 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.316173 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.317535 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.317545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.317915 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.318841 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:03.318873 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:03.322537 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:03.326416 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:03.326426 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:03.326443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:03.327054 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:03.327506 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:03.327510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:03.327512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:03.327893 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:03.328526 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:03.328534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:03.328950 (+ 416us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.329268 (+ 318us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.329973 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"thread_start_us":798,"threads_started":1}
I20260809 05:21:03.333277 24040 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34885
I20260809 05:21:03.333700 24040 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:03.334848 24040 heartbeater.cc:511] Master 127.22.1.126:34885 requested a full tablet report, sending...
I20260809 05:21:03.336573 23366 ts_manager.cc:194] Registered new tserver with Master: fa8b207bd3944daa877a42823c626a65 (127.22.1.67:45927)
I20260809 05:21:03.337639 23366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:39757
I20260809 05:21:03.347498 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:03.348469 23631 heartbeater.cc:503] Master 127.22.1.126:34885 was elected leader, sending a full tablet report...
I20260809 05:21:03.375600 23633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.358293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.22.1.126:34885)
0809 05:21:03.358519 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.358523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.359188 (+ 665us) server_negotiation.cc:407] Connection header received
0809 05:21:03.359414 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.359419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.359516 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.359604 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.361844 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.362827 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.364702 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.365362 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.365499 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.369053 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.369059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.369063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.369101 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.372199 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.373015 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.373021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.373025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.373325 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.373758 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.373764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.373766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.374240 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.374518 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.374927 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.375183 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:21:03.385465 23366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:03.397317 23366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:03.433220 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.417650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58626 (local address 127.22.1.66:40061)
0809 05:21:03.420333 (+ 2683us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.420344 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.421328 (+ 984us) server_negotiation.cc:407] Connection header received
0809 05:21:03.421752 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.421761 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.422030 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.422324 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:03.423791 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.426085 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.429430 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.431567 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.431703 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.431931 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.432517 (+ 586us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.432702 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2477,"thread_start_us":2259,"threads_started":1}
I20260809 05:21:03.433408 24063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.417732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:40061 (local address 127.0.0.1:58626)
0809 05:21:03.421208 (+ 3476us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.421228 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.421494 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.422788 (+ 1294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.422797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.422832 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:03.423553 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.423567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.426298 (+ 2731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.426304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.429233 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.429245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.431835 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.431843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.432151 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.432260 (+ 109us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.432432 (+ 172us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.432647 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3262,"mutex_wait_us":31,"thread_start_us":3012,"threads_started":1}
I20260809 05:21:03.438999 24060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.417698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45927 (local address 127.0.0.1:47662)
0809 05:21:03.418841 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.418886 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.419173 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.429194 (+ 10021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.429199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.429228 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:03.430656 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.430673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.433528 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.433534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.436187 (+ 2653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.436201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.437882 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.437888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.438084 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.438118 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.438208 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.438351 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":1404,"threads_started":1}
I20260809 05:21:03.438977 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.418728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47662 (local address 127.22.1.67:45927)
0809 05:21:03.428186 (+ 9458us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.428195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.428255 (+ 60us) server_negotiation.cc:407] Connection header received
0809 05:21:03.428351 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.428360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.428647 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.429431 (+ 784us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:03.430954 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.433336 (+ 2382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.436363 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.437644 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.437763 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.438089 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.438262 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.438476 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9212,"thread_start_us":8898,"threads_started":1}
I20260809 05:21:03.439080 23731 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6851227354982b131c5859f5bcc00 (DEFAULT_TABLE table=test-table [id=cba38166d65c41ad8a70ea5b8659d32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:03.441286 23731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6851227354982b131c5859f5bcc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:03.442314 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.421421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:39393 (local address 127.0.0.1:56540)
0809 05:21:03.430965 (+ 9544us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.430985 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.431144 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.432023 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.432029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.432055 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:03.432967 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.432982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.435728 (+ 2746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.435733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.438328 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.438344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.441113 (+ 2769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.441122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.441406 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.441450 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.441530 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.441655 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9268,"mutex_wait_us":3,"thread_start_us":8904,"threads_started":1}
I20260809 05:21:03.443362 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.422032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.22.1.65:39393)
0809 05:21:03.428804 (+ 6772us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.428814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.431279 (+ 2465us) server_negotiation.cc:407] Connection header received
0809 05:21:03.431385 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.431392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.431680 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.431903 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:03.433338 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.435503 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.438532 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.440288 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.440420 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.440845 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.442705 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.442888 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6548,"thread_start_us":6832,"threads_started":1}
I20260809 05:21:03.443238 23932 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6851227354982b131c5859f5bcc00 (DEFAULT_TABLE table=test-table [id=cba38166d65c41ad8a70ea5b8659d32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:03.444902 23932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6851227354982b131c5859f5bcc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:03.447644 23540 tablet_service.cc:1467] Processing CreateTablet for tablet 6ff6851227354982b131c5859f5bcc00 (DEFAULT_TABLE table=test-table [id=cba38166d65c41ad8a70ea5b8659d32e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:03.449359 23540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ff6851227354982b131c5859f5bcc00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:03.458354 24067 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Bootstrap starting.
I20260809 05:21:03.463207 24067 tablet_bootstrap.cc:654] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:03.465021 24068 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Bootstrap starting.
I20260809 05:21:03.465847 24067 log.cc:824] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:03.469792 24070 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Bootstrap starting.
I20260809 05:21:03.469863 24067 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: No bootstrap required, opened a new log
I20260809 05:21:03.469986 24068 tablet_bootstrap.cc:654] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:03.470167 24067 ts_tablet_manager.cc:1397] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260809 05:21:03.471540 24068 log.cc:824] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:03.474496 24070 tablet_bootstrap.cc:654] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:03.475395 24068 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: No bootstrap required, opened a new log
I20260809 05:21:03.475746 24068 ts_tablet_manager.cc:1397] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260809 05:21:03.476939 24070 log.cc:824] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:03.481043 24070 tablet_bootstrap.cc:492] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: No bootstrap required, opened a new log
I20260809 05:21:03.481505 24070 ts_tablet_manager.cc:1397] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20260809 05:21:03.481714 24067 raft_consensus.cc:348] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.482237 24067 raft_consensus.cc:374] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:03.482388 24067 raft_consensus.cc:729] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3903a2fc1a45d383958e75169917c4, State: Initialized, Role: FOLLOWER
I20260809 05:21:03.482982 24067 consensus_queue.cc:260] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.484858 23828 heartbeater.cc:503] Master 127.22.1.126:34885 was elected leader, sending a full tablet report...
I20260809 05:21:03.485564 24067 ts_tablet_manager.cc:1428] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260809 05:21:03.491044 24068 raft_consensus.cc:348] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.491741 24068 raft_consensus.cc:374] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:03.491925 24068 raft_consensus.cc:729] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8b207bd3944daa877a42823c626a65, State: Initialized, Role: FOLLOWER
I20260809 05:21:03.492761 24068 consensus_queue.cc:260] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.494328 24040 heartbeater.cc:503] Master 127.22.1.126:34885 was elected leader, sending a full tablet report...
I20260809 05:21:03.495304 24068 ts_tablet_manager.cc:1428] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Time spent starting tablet: real 0.019s user 0.011s sys 0.010s
I20260809 05:21:03.495342 24070 raft_consensus.cc:348] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.495929 24070 raft_consensus.cc:374] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:03.496069 24070 raft_consensus.cc:729] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a019f3c520e4051902d86411fdd1ee5, State: Initialized, Role: FOLLOWER
I20260809 05:21:03.496685 24070 consensus_queue.cc:260] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.498677 24070 ts_tablet_manager.cc:1428] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Time spent starting tablet: real 0.017s user 0.011s sys 0.006s
I20260809 05:21:03.547928 24075 raft_consensus.cc:482] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:03.548293 24075 raft_consensus.cc:504] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.551334 24075 leader_election.cc:288] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3903a2fc1a45d383958e75169917c4 (127.22.1.66:40061), 4a019f3c520e4051902d86411fdd1ee5 (127.22.1.65:39393)
W20260809 05:21:03.554797 24041 tablet.cc:2369] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:03.556576 23829 tablet.cc:2369] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:03.560859 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.551862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:45121 (local address 127.22.1.65:39393)
0809 05:21:03.552078 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.552084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.552643 (+ 559us) server_negotiation.cc:407] Connection header received
0809 05:21:03.552819 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.552825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.552906 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.553025 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:03.553489 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.554978 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.558019 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.559687 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.559883 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.560032 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.560342 (+ 310us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.560497 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:21:03.560875 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.551584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:39393 (local address 127.22.1.67:45121)
0809 05:21:03.552518 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.552544 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.552690 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.553032 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.553038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.553065 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:03.553345 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.553362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.555182 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.555187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.557837 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.557856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.559889 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.559894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.560120 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.560164 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.560277 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.560437 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":33,"thread_start_us":451,"threads_started":1}
I20260809 05:21:03.560969 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.552000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:48821 (local address 127.22.1.66:40061)
0809 05:21:03.552260 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.552267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.552652 (+ 385us) server_negotiation.cc:407] Connection header received
0809 05:21:03.552883 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.552888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.552957 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.553045 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:03.553735 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.554924 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.558208 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.559844 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.560048 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.560130 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.560566 (+ 436us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.560707 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260809 05:21:03.560989 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.551554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:40061 (local address 127.22.1.67:48821)
0809 05:21:03.552556 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:03.552570 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:03.552761 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:03.553055 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:03.553059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:03.553076 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:03.553544 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.553554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.555182 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.555187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.558020 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:03.558033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.560158 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.560164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:03.560386 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.560417 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 05:21:03.560497 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 05:21:03.560649 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":31,"spinlock_wait_cycles":23936}
I20260809 05:21:03.562546 23765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6851227354982b131c5859f5bcc00" candidate_uuid: "fa8b207bd3944daa877a42823c626a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3903a2fc1a45d383958e75169917c4" is_pre_election: true
I20260809 05:21:03.562548 23569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6851227354982b131c5859f5bcc00" candidate_uuid: "fa8b207bd3944daa877a42823c626a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a019f3c520e4051902d86411fdd1ee5" is_pre_election: true
I20260809 05:21:03.563490 23765 raft_consensus.cc:2454] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8b207bd3944daa877a42823c626a65 in term 0.
I20260809 05:21:03.564664 23569 raft_consensus.cc:2454] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa8b207bd3944daa877a42823c626a65 in term 0.
I20260809 05:21:03.564684 23863 leader_election.cc:302] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3903a2fc1a45d383958e75169917c4, fa8b207bd3944daa877a42823c626a65; no voters:
I20260809 05:21:03.565212 24075 raft_consensus.cc:2781] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:03.565354 24075 raft_consensus.cc:482] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:03.565513 24075 raft_consensus.cc:3037] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 0 FOLLOWER]: Advancing to term 1
W20260809 05:21:03.568548 23632 tablet.cc:2369] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:03.568962 24075 raft_consensus.cc:504] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.569784 24075 leader_election.cc:288] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [CANDIDATE]: Term 1 election: Requested vote from peers bd3903a2fc1a45d383958e75169917c4 (127.22.1.66:40061), 4a019f3c520e4051902d86411fdd1ee5 (127.22.1.65:39393)
I20260809 05:21:03.570291 23765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6851227354982b131c5859f5bcc00" candidate_uuid: "fa8b207bd3944daa877a42823c626a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3903a2fc1a45d383958e75169917c4"
I20260809 05:21:03.570364 23569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ff6851227354982b131c5859f5bcc00" candidate_uuid: "fa8b207bd3944daa877a42823c626a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a019f3c520e4051902d86411fdd1ee5"
I20260809 05:21:03.570537 23765 raft_consensus.cc:3037] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:03.570628 23569 raft_consensus.cc:3037] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:03.573670 23569 raft_consensus.cc:2454] T 6ff6851227354982b131c5859f5bcc00 P 4a019f3c520e4051902d86411fdd1ee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8b207bd3944daa877a42823c626a65 in term 1.
I20260809 05:21:03.573734 23765 raft_consensus.cc:2454] T 6ff6851227354982b131c5859f5bcc00 P bd3903a2fc1a45d383958e75169917c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa8b207bd3944daa877a42823c626a65 in term 1.
I20260809 05:21:03.574162 23866 leader_election.cc:302] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a019f3c520e4051902d86411fdd1ee5, fa8b207bd3944daa877a42823c626a65; no voters:
I20260809 05:21:03.574556 24075 raft_consensus.cc:2781] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:03.575207 24075 raft_consensus.cc:686] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [term 1 LEADER]: Becoming Leader. State: Replica: fa8b207bd3944daa877a42823c626a65, State: Running, Role: LEADER
I20260809 05:21:03.575925 24075 consensus_queue.cc:237] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } }
I20260809 05:21:03.582548 23365 catalog_manager.cc:5495] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa8b207bd3944daa877a42823c626a65 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "fa8b207bd3944daa877a42823c626a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3903a2fc1a45d383958e75169917c4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 40061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa8b207bd3944daa877a42823c626a65" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a019f3c520e4051902d86411fdd1ee5" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 39393 } health_report { overall_health: UNKNOWN } } }
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:03.642025 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.619448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.22.1.65:39393)
0809 05:21:03.619626 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.619631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.621106 (+ 1475us) server_negotiation.cc:407] Connection header received
0809 05:21:03.621310 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.621315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.621481 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.621586 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.624479 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.625684 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.627188 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.627793 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.627916 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.634015 (+ 6099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.634034 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.634054 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.634140 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.638365 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.639141 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.639150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.639152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.639405 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.639898 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.639902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.639904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.640465 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.640868 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.641264 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.641502 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:03.701491 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.682165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.22.1.65:39393)
0809 05:21:03.682403 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.682410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.682898 (+ 488us) server_negotiation.cc:407] Connection header received
0809 05:21:03.683080 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.683084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.683152 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.683237 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.685165 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.686501 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.688352 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.689103 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.689275 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.694640 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.694647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.694651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.694695 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.698373 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.699048 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.699053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.699055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.699273 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.699778 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.699782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.699784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.700163 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.700487 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.700769 (+ 282us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.701027 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:03.761050 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.740244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.22.1.65:39393)
0809 05:21:03.740519 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.740527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.741031 (+ 504us) server_negotiation.cc:407] Connection header received
0809 05:21:03.741249 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.741253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.741322 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.741427 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.744139 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.744854 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.746839 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.747285 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.747383 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.753449 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.753459 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.753465 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.753517 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.757998 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.758752 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.758757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.758759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.758892 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.759258 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.759261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.759264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.759605 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.759841 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.760238 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.760596 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 05:21:03.763604 23507 tablet_service.cc:2908] Rejecting scan request for tablet 6ff6851227354982b131c5859f5bcc00: Uninitialized: clean time has not yet been initialized
I20260809 05:21:03.768112 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23451
I20260809 05:21:03.783335 24066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.769101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.22.1.65:39393)
0809 05:21:03.769272 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.769276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.769745 (+ 469us) server_negotiation.cc:407] Connection header received
0809 05:21:03.769873 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.769877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.769983 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.770065 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.771175 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.771938 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.773077 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.773859 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.773999 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.777096 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.777102 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.777106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.777185 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.780595 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.781425 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.781429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.781431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.781611 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.781991 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.781994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.781996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.782290 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.782609 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.782803 (+ 194us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.782986 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:21:03.784119 23622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:03.882349 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23451
I20260809 05:21:03.906620 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23637
Aug 09 05:21:03 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:03.920751 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:03.907390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.22.1.66:40061)
0809 05:21:03.907588 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:03.907593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:03.907612 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:21:03.907744 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:03.907748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:03.907840 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:03.907922 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:03.909011 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.909801 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.910993 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:03.911374 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:03.911461 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:03.915237 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:03.915251 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:03.915269 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:03.915305 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:03.918049 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.918662 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.918668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.918670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.918799 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:03.919155 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:03.919160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:03.919162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:03.919626 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:03.919938 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:03.920208 (+ 270us) server_negotiation.cc:299] Negotiation successful
0809 05:21:03.920415 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:21:03.921404 23821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:04.008662 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23637
I20260809 05:21:04.031164 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23834
W20260809 05:21:04.030938 23863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.66:40061 (error 108)
I20260809 05:21:04.032971 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:04.032402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:40061 (local address 127.22.1.67:52579)
0809 05:21:04.032589 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:04.032767 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.66:40061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260809 05:21:04.033357 23863 consensus_peers.cc:487] T 6ff6851227354982b131c5859f5bcc00 P fa8b207bd3944daa877a42823c626a65 -> Peer bd3903a2fc1a45d383958e75169917c4 (127.22.1.66:40061): Couldn't send request to peer bd3903a2fc1a45d383958e75169917c4. Status: Network error: Client connection negotiation failed: client connection to 127.22.1.66:40061: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 09 05:21:04 dist-test-slave-rmhg krb5kdc[23304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:04.046084 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:04.032001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.22.1.67:45927)
0809 05:21:04.032423 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:04.032428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:04.032451 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:21:04.032518 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:04.032523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:04.032599 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:04.032776 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:04.034215 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:04.034911 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:04.036099 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:04.036492 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:04.036581 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:04.040426 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:04.040438 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:04.040455 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:04.040489 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:04.043338 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:04.043997 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:04.044005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:04.044007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:04.044165 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:04.044551 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:04.044555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:04.044557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:04.044902 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:04.045202 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:04.045511 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 05:21:04.045731 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":537,"threads_started":1}
I20260809 05:21:04.046686 24033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:04.135406 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23834
I20260809 05:21:04.135524 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:04.135110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:39393 (local address 127.22.1.67:40651)
0809 05:21:04.135294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:04.135384 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:39393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 05:21:04.155984 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23322
I20260809 05:21:04.157184 23409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:04.263482 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23322
2026-08-09T05:21:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5601 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:04 dist-test-slave-rmhg krb5kdc[24109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:04 dist-test-slave-rmhg krb5kdc[24109](info): set up 2 sockets
Aug 09 05:21:04 dist-test-slave-rmhg krb5kdc[24109](info): commencing operation
krb5kdc: starting...
W20260809 05:21:05.439378 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:05 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:05.632138 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35489
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:05.958724 24133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:05.959014 24133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:05.959143 24133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:05.966454 24133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:05.966578 24133 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 05:21:05.966625 24133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:05.966712 24133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:05.966771 24133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:05.966858 24133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:05.977339 24133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:05.978780 24133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:05.980237 24133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:05.992494 24138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:05.992451 24142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:05.992538 24139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:05.993638 24133 server_base.cc:1034] running on GCE node
I20260809 05:21:05.994432 24133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:05.995918 24133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:05.997171 24133 hybrid_clock.cc:648] HybridClock initialized: now 1786252865997087 us; error 41 us; skew 500 ppm
I20260809 05:21:05.997524 24133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:05 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252865, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:06.000661 24133 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:06.002233 24133 webserver.cc:460] Webserver started at http://127.22.1.126:46787/ using document root <none> and password file <none>
I20260809 05:21:06.002962 24133 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:06.003082 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:06.003412 24133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:06.006577 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "9482b5c8f97d4a1d90e14d0e6f820f53"
format_stamp: "Formatted at 2026-08-09 05:21:05 on dist-test-slave-rmhg"
server_key: "32c39fefb5bb2f71f33fa101959a2925"
server_key_iv: "e490eaffd7efc70bcaf959218051b7ea"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.007382 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "9482b5c8f97d4a1d90e14d0e6f820f53"
format_stamp: "Formatted at 2026-08-09 05:21:05 on dist-test-slave-rmhg"
server_key: "32c39fefb5bb2f71f33fa101959a2925"
server_key_iv: "e490eaffd7efc70bcaf959218051b7ea"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.012902 24133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:06.016665 24152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:06.018054 24133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:06.018221 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "9482b5c8f97d4a1d90e14d0e6f820f53"
format_stamp: "Formatted at 2026-08-09 05:21:05 on dist-test-slave-rmhg"
server_key: "32c39fefb5bb2f71f33fa101959a2925"
server_key_iv: "e490eaffd7efc70bcaf959218051b7ea"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.018428 24133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:06.057464 24133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:06.061685 24133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:06.061947 24133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:06.083011 24133 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35489
I20260809 05:21:06.083032 24244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35489 every 8 connection(s)
I20260809 05:21:06.084460 24133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:06.088418 24246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:06.091972 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24133
I20260809 05:21:06.092167 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:06.092819 22533 external_mini_cluster.cc:1428] Setting key 18e9b5c59f91055bd9158b2bbfb0030f
I20260809 05:21:06.099745 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Bootstrap starting.
I20260809 05:21:06.103448 24246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:06.104607 24246 log.cc:824] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:06.107972 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: No bootstrap required, opened a new log
Aug 09 05:21:06 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:06.116854 24246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } }
I20260809 05:21:06.117270 24246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:06.117379 24246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9482b5c8f97d4a1d90e14d0e6f820f53, State: Initialized, Role: FOLLOWER
I20260809 05:21:06.117786 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:06.094937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.22.1.126:35489)
0809 05:21:06.095743 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:06.095755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:06.095802 (+ 47us) server_negotiation.cc:407] Connection header received
0809 05:21:06.098192 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:06.098217 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:06.098642 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:06.098968 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:06.100773 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.102167 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.103859 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.104591 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.104723 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:06.109340 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:06.109348 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:06.109363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:06.109399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:06.113502 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.114204 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.114214 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.114223 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.114402 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.114782 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.114788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.114792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.115391 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:06.115692 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:06.116154 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 05:21:06.116578 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":343,"threads_started":1}
I20260809 05:21:06.118068 24246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } }
I20260809 05:21:06.118374 24246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:06.118530 24246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:06.118716 24246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:06.121349 24246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } }
I20260809 05:21:06.121845 24246 leader_election.cc:302] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9482b5c8f97d4a1d90e14d0e6f820f53; no voters:
I20260809 05:21:06.122396 24246 leader_election.cc:288] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:06.122556 24254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:06.124037 24254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [term 1 LEADER]: Becoming Leader. State: Replica: 9482b5c8f97d4a1d90e14d0e6f820f53, State: Running, Role: LEADER
I20260809 05:21:06.124619 24246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:06.124737 24254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } }
I20260809 05:21:06.127399 24256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9482b5c8f97d4a1d90e14d0e6f820f53. Latest consensus state: current_term: 1 leader_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } } }
I20260809 05:21:06.127838 24256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:06.127415 24255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9482b5c8f97d4a1d90e14d0e6f820f53" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35489 } } }
I20260809 05:21:06.128206 24255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:06.130688 24265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:06.135596 24265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:06.144229 24265 catalog_manager.cc:1347] Generated new cluster ID: e3fc2455b0234c13be7c164bb79cd1ef
I20260809 05:21:06.144366 24265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:06.185137 24265 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:06.187052 24265 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:06.199091 24265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Generated new TSK 0
I20260809 05:21:06.199975 24265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:06.256479 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35489
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:06.605191 24295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:06.605518 24295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:06.605664 24295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:06.613252 24295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:06.613390 24295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:06.613525 24295 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:06.613641 24295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:06.623966 24295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:06.625469 24295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:06.627107 24295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:06.641065 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:06.640970 24308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:06.641018 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:06.641903 24295 server_base.cc:1034] running on GCE node
I20260809 05:21:06.642566 24295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:06.643569 24295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:06.644840 24295 hybrid_clock.cc:648] HybridClock initialized: now 1786252866644749 us; error 66 us; skew 500 ppm
I20260809 05:21:06.645244 24295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:06 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252866, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:06.648821 24295 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:06.650527 24295 webserver.cc:460] Webserver started at http://127.22.1.65:45473/ using document root <none> and password file <none>
I20260809 05:21:06.651305 24295 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:06.651419 24295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:06.651772 24295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:06.655105 24295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "5473333fe553408fba467a1cbeb66365"
format_stamp: "Formatted at 2026-08-09 05:21:06 on dist-test-slave-rmhg"
server_key: "df7226b9ad7c7af28ebb4069d7d647f5"
server_key_iv: "01f0d3970508ad50f0f37fede22c6797"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.656021 24295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "5473333fe553408fba467a1cbeb66365"
format_stamp: "Formatted at 2026-08-09 05:21:06 on dist-test-slave-rmhg"
server_key: "df7226b9ad7c7af28ebb4069d7d647f5"
server_key_iv: "01f0d3970508ad50f0f37fede22c6797"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.662041 24295 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260809 05:21:06.666069 24316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:06.667634 24295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:06.667809 24295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "5473333fe553408fba467a1cbeb66365"
format_stamp: "Formatted at 2026-08-09 05:21:06 on dist-test-slave-rmhg"
server_key: "df7226b9ad7c7af28ebb4069d7d647f5"
server_key_iv: "01f0d3970508ad50f0f37fede22c6797"
server_key_version: "encryptionkey@0"
I20260809 05:21:06.668032 24295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:06.684418 24295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:06.689059 24295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:06.689359 24295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:06.692375 24295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:06.692497 24295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:06.692636 24295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:06.692714 24295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:21:06 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252866, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:06.717574 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:06.693136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56159 (local address 127.22.1.126:35489)
0809 05:21:06.693646 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:06.693650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:06.694976 (+ 1326us) server_negotiation.cc:407] Connection header received
0809 05:21:06.697191 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:06.697199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:06.697294 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:06.697397 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:06.699780 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.700963 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.703596 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.704171 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.704320 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:06.709843 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:06.709849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:06.709853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:06.709922 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:06.713180 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.714687 (+ 1507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.714694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.714696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.714840 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.715426 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.715429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.715431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.715715 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:06.715828 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:06.716949 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0809 05:21:06.717198 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":244,"threads_started":1}
I20260809 05:21:06.719744 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:06.693902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.65:56159)
0809 05:21:06.694797 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:06.694844 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:06.696853 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:06.697685 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:06.697694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:06.698552 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:06.699535 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:06.699563 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.701242 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.701248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:06.703419 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:06.703434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.704518 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.704529 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:06.704892 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:06.705888 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:06.705927 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:06.709500 (+ 3573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:06.713390 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:06.713402 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:06.713433 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:06.714495 (+ 1062us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:06.714954 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:06.714960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:06.714964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:06.715317 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:06.715858 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:06.715869 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:06.716472 (+ 603us) client_negotiation.cc:769] Sending connection context
0809 05:21:06.716852 (+ 380us) client_negotiation.cc:240] Negotiation successful
0809 05:21:06.717732 (+ 880us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":384,"threads_started":1}
I20260809 05:21:06.738121 24295 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:41577
I20260809 05:21:06.738139 24495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:41577 every 8 connection(s)
I20260809 05:21:06.739521 24295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:06.740347 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:06.726582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.65:45563)
0809 05:21:06.727280 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:06.727300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:06.727458 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:06.727794 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:06.727800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:06.728300 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:06.728805 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:06.728822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.730101 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.730107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:06.731792 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:06.731809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.732546 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.732553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:06.732729 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:06.733453 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:06.733474 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:06.734223 (+ 749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:06.737692 (+ 3469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:06.737699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:06.737703 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:06.738195 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:06.738638 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:06.738641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:06.738644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:06.738778 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:06.739322 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:06.739327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:06.739527 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 05:21:06.739662 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 05:21:06.739964 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":549,"threads_started":1}
I20260809 05:21:06.741264 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:06.727090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:45563 (local address 127.22.1.126:35489)
0809 05:21:06.727280 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:06.727287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:06.727364 (+ 77us) server_negotiation.cc:407] Connection header received
0809 05:21:06.727584 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:06.727589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:06.727657 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:06.727737 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:06.728994 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.729977 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.731937 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:06.732362 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:06.732475 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:06.734425 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:06.734432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:06.734438 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:06.734482 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:06.737548 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.738368 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.738375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.738378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.738520 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:06.738891 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:06.738896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:06.738898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:06.739198 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:06.739354 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:06.740728 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0809 05:21:06.740982 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:21:06.742229 24497 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35489
I20260809 05:21:06.742468 24497 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:06.743440 24497 heartbeater.cc:511] Master 127.22.1.126:35489 requested a full tablet report, sending...
I20260809 05:21:06.745894 24179 ts_manager.cc:194] Registered new tserver with Master: 5473333fe553408fba467a1cbeb66365 (127.22.1.65:41577)
I20260809 05:21:06.747385 24179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:56159
I20260809 05:21:06.748425 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24295
I20260809 05:21:06.748654 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:06.749361 22533 external_mini_cluster.cc:1428] Setting key f5580c93875650d8a4916a43fdfc6ddf
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:06.792408 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35489
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:07.123029 24502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:07.123399 24502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:07.123549 24502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:07.131075 24502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:07.131207 24502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:07.131330 24502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:07.131453 24502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:07.141680 24502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:07.143090 24502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:07.144572 24502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:07.157882 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:07.157913 24519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:07.157882 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:07.158851 24502 server_base.cc:1034] running on GCE node
I20260809 05:21:07.159488 24502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:07.160481 24502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:07.161731 24502 hybrid_clock.cc:648] HybridClock initialized: now 1786252867161635 us; error 55 us; skew 500 ppm
I20260809 05:21:07.162067 24502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:07.165268 24502 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:07.166690 24502 webserver.cc:460] Webserver started at http://127.22.1.66:35921/ using document root <none> and password file <none>
I20260809 05:21:07.167387 24502 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:07.167488 24502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:07.167804 24502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:07.170872 24502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "8222712b24eb48548de5cee900798905"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "c2bcd39063041e46d01584bc485dd263"
server_key_iv: "57c0a282ce9adeeafbc612129f29c287"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.171705 24502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "8222712b24eb48548de5cee900798905"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "c2bcd39063041e46d01584bc485dd263"
server_key_iv: "57c0a282ce9adeeafbc612129f29c287"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.176899 24502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:07.180326 24529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:07.181789 24502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:07.181967 24502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "8222712b24eb48548de5cee900798905"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "c2bcd39063041e46d01584bc485dd263"
server_key_iv: "57c0a282ce9adeeafbc612129f29c287"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.182178 24502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:07.208321 24502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:07.213001 24502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:07.213316 24502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:07.216140 24502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:07.216252 24502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:07.216384 24502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:07.216455 24502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:07.240127 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.216902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:59355 (local address 127.22.1.126:35489)
0809 05:21:07.217082 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.217087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.218748 (+ 1661us) server_negotiation.cc:407] Connection header received
0809 05:21:07.220919 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.220925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.221046 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.221197 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:07.223701 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.224718 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.227331 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.227889 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.228037 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.233292 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:07.233299 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:07.233304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:07.233348 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:07.236272 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.237591 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.237598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.237601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.237784 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.238319 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.238324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.238326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.238626 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:07.238767 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.239530 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.239769 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:21:07.241281 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.217722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.66:59355)
0809 05:21:07.218587 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.218625 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.220604 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.221469 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.221482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.222527 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:07.223499 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.223536 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.224885 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.224892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.227179 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.227192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.228151 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.228160 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.228404 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.229139 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:07.229198 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:07.232939 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:07.236453 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.236466 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.236488 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.237364 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.237940 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.237946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.237948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.238182 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.238763 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:07.238771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:07.239144 (+ 373us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.239396 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.239969 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
I20260809 05:21:07.261497 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.247035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:56565 (local address 127.22.1.126:35489)
0809 05:21:07.247248 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.247254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.247846 (+ 592us) server_negotiation.cc:407] Connection header received
0809 05:21:07.248036 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.248040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.248103 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.248181 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:07.249591 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.250541 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.252600 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.253018 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.253112 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.255563 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:07.255567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:07.255571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:07.255600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:07.258749 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.259583 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.259587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.259589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.259713 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.260093 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.260097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.260099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.260447 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:07.260570 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.260987 (+ 417us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.261175 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:21:07.261852 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.247006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.66:56565)
0809 05:21:07.247727 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.247754 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.247912 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.248227 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.248233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.248923 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:07.249390 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.249408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.250651 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.250658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.252447 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.252466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.253210 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.253218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.253408 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.254056 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:07.254080 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:07.255365 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:07.258907 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.258914 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.258918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.259478 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.259809 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.259814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.259818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.259975 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.260544 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:07.260549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:07.260764 (+ 215us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.260926 (+ 162us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.261342 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":378,"threads_started":1}
I20260809 05:21:07.272994 24502 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:42163
I20260809 05:21:07.273029 24696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:42163 every 8 connection(s)
I20260809 05:21:07.274217 24502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:07.276135 24697 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35489
I20260809 05:21:07.276338 24697 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:07.276957 24697 heartbeater.cc:511] Master 127.22.1.126:35489 requested a full tablet report, sending...
I20260809 05:21:07.278264 24179 ts_manager.cc:194] Registered new tserver with Master: 8222712b24eb48548de5cee900798905 (127.22.1.66:42163)
I20260809 05:21:07.279062 24179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:59355
I20260809 05:21:07.282878 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24502
I20260809 05:21:07.283041 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:07.283465 22533 external_mini_cluster.cc:1428] Setting key e896f9ba492e346cfa3fae966277f849
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:07.326601 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35489
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:07.652971 24704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:07.653328 24704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:07.653484 24704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:07.661116 24704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:07.661294 24704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:07.661391 24704 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:07.661474 24704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:07.671679 24704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:07.673313 24704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:07.674746 24704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:07.687641 24711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:07.687580 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:07.687616 24712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:07.688751 24704 server_base.cc:1034] running on GCE node
I20260809 05:21:07.689428 24704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:07.690387 24704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:07.691624 24704 hybrid_clock.cc:648] HybridClock initialized: now 1786252867691560 us; error 42 us; skew 500 ppm
I20260809 05:21:07.691922 24704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:07.695133 24704 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:07.696601 24704 webserver.cc:460] Webserver started at http://127.22.1.67:46415/ using document root <none> and password file <none>
I20260809 05:21:07.697337 24704 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:07.697448 24704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:07.697768 24704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:07.700707 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "984d7ddadbca48878b5bc6495277635b"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "70f8f552d001cd7eaf8700afeeb424b7"
server_key_iv: "cd2febec1cdded532e04bdfe8374c1bc"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.701531 24704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "984d7ddadbca48878b5bc6495277635b"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "70f8f552d001cd7eaf8700afeeb424b7"
server_key_iv: "cd2febec1cdded532e04bdfe8374c1bc"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.707033 24704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:07.710582 24726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:07.711951 24704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:07.712121 24704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "984d7ddadbca48878b5bc6495277635b"
format_stamp: "Formatted at 2026-08-09 05:21:07 on dist-test-slave-rmhg"
server_key: "70f8f552d001cd7eaf8700afeeb424b7"
server_key_iv: "cd2febec1cdded532e04bdfe8374c1bc"
server_key_version: "encryptionkey@0"
I20260809 05:21:07.712358 24704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:07.730682 24704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:07.734828 24704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:07.735095 24704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:07.738058 24704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:07.738166 24704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:07.738358 24704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:07.738484 24704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:07.750450 24497 heartbeater.cc:503] Master 127.22.1.126:35489 was elected leader, sending a full tablet report...
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:07.762320 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.738586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47417 (local address 127.22.1.126:35489)
0809 05:21:07.738784 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.738790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.742077 (+ 3287us) server_negotiation.cc:407] Connection header received
0809 05:21:07.745036 (+ 2959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.745043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.745171 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.745260 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:07.747740 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.748504 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.751507 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.751936 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.752071 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.756293 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:07.756299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:07.756303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:07.756338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:07.759134 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.759889 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.759894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.759896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.760061 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.760477 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.760481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.760483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.760754 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:07.760899 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.761704 (+ 805us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.761947 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:21:07.763795 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.739256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.67:47417)
0809 05:21:07.741876 (+ 2620us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.741937 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.744622 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.745829 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.745842 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.746630 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:07.747539 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.747571 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.748693 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.748698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.751368 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.751382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.752207 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.752215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.752552 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.753390 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:07.753414 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:07.756052 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:07.759277 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.759286 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.759305 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.759756 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.760170 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.760176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.760180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.760363 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.760881 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:07.760889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:07.761312 (+ 423us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.761585 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.762185 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2221,"thread_start_us":1997,"threads_started":1}
I20260809 05:21:07.790812 24704 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:33633
I20260809 05:21:07.790868 24902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:33633 every 8 connection(s)
I20260809 05:21:07.792403 24704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:07.792708 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.770584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:53993 (local address 127.22.1.126:35489)
0809 05:21:07.770751 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.770755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.777009 (+ 6254us) server_negotiation.cc:407] Connection header received
0809 05:21:07.777232 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.777238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.777332 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.777446 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:07.779071 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.780191 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.782364 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.782881 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.783020 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.785921 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:07.785928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:07.785933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:07.786002 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:07.789620 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.790672 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.790679 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.790683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.790854 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.791279 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.791285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.791288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.791571 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:07.791704 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.792139 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.792359 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 05:21:07.792819 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.775875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.22.1.67:53993)
0809 05:21:07.776862 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.776889 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.777064 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.777507 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.777512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.778266 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:07.778880 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.778899 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.780380 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.780385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.782217 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.782232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.783207 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.783216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.783414 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.784120 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:07.784146 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:07.785696 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:07.789794 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.789801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.789806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.790512 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.791004 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:07.791009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:07.791012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:07.791156 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:07.791690 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:07.791695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:07.791878 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.792002 (+ 124us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.792350 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":711,"threads_started":1}
I20260809 05:21:07.795341 24905 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35489
I20260809 05:21:07.795634 24905 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:07.796263 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24704
I20260809 05:21:07.796448 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:07.796478 24905 heartbeater.cc:511] Master 127.22.1.126:35489 requested a full tablet report, sending...
I20260809 05:21:07.797014 22533 external_mini_cluster.cc:1428] Setting key 5ad2df78fa2be75485ad2a85c49e0e9d
I20260809 05:21:07.797972 24179 ts_manager.cc:194] Registered new tserver with Master: 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633)
I20260809 05:21:07.799014 24179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:47417
I20260809 05:21:07.800258 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 05:21:07 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:07.850215 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.835996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.22.1.126:35489)
0809 05:21:07.836242 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.836249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.836397 (+ 148us) server_negotiation.cc:407] Connection header received
0809 05:21:07.836597 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.836603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.836674 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.836764 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:07.838266 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.839024 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.840333 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.840774 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.840890 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.844909 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:07.844914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:07.844918 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:07.844964 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:07.847539 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.848205 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.848210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.848212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.848356 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:07.848748 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:07.848752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:07.848754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:07.849045 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:07.849309 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.849626 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.849858 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260809 05:21:07.854630 24178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:07.861665 24178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:07.894753 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.874526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.22.1.67:33633)
0809 05:21:07.886277 (+ 11751us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.886283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.886316 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:21:07.886409 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.886416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.886802 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.887014 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:07.888003 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.889585 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.891973 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893400 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.893523 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.893755 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.894306 (+ 551us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.894449 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11589,"thread_start_us":11414,"threads_started":1}
I20260809 05:21:07.894742 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.875502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.22.1.66:42163)
0809 05:21:07.885646 (+ 10144us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.885656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.885692 (+ 36us) server_negotiation.cc:407] Connection header received
0809 05:21:07.885807 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.885816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.886127 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.886373 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:07.887139 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.888734 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.891714 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893083 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.893301 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.893528 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.894242 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.894426 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9939,"thread_start_us":9789,"threads_started":1}
I20260809 05:21:07.894802 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.874883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42163 (local address 127.0.0.1:42178)
0809 05:21:07.884845 (+ 9962us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.884870 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.885043 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.886407 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.886416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.886451 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:07.886899 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.886916 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.888892 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.888898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.891522 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.891542 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893648 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.893655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.893853 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.894023 (+ 170us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.894171 (+ 148us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.894374 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9479,"mutex_wait_us":87,"thread_start_us":9217,"threads_started":1}
I20260809 05:21:07.895881 24921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.874335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33633 (local address 127.0.0.1:46824)
0809 05:21:07.875089 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.875141 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.875301 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.887311 (+ 12010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.887316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.887341 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:07.887735 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.887748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.889753 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.889759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.891833 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.891843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893903 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.893911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.894080 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.894104 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.894192 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.895541 (+ 1349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":46,"thread_start_us":331,"threads_started":1}
I20260809 05:21:07.897925 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.875460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41577 (local address 127.0.0.1:41970)
0809 05:21:07.890280 (+ 14820us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:07.890300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:07.890429 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:07.890930 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:07.890934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:07.890968 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:07.891351 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.891360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893334 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.893339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.895493 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:07.895506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.897123 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.897130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:07.897333 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.897366 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 05:21:07.897461 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:21:07.897585 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14693,"mutex_wait_us":1,"thread_start_us":13712,"threads_started":1}
I20260809 05:21:07.897969 24927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:07.875793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.22.1.65:41577)
0809 05:21:07.889376 (+ 13583us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:07.889384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:07.890407 (+ 1023us) server_negotiation.cc:407] Connection header received
0809 05:21:07.890578 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:07.890584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:07.890760 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:07.890923 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:07.891518 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.893133 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.895639 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:07.896909 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:07.897054 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:07.897287 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:07.897532 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 05:21:07.897690 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13411,"thread_start_us":13586,"threads_started":1}
I20260809 05:21:07.898841 24597 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (DEFAULT_TABLE table=test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:07.899140 24799 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (DEFAULT_TABLE table=test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:07.900034 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:07.900307 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:07.904671 24396 tablet_service.cc:1467] Processing CreateTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (DEFAULT_TABLE table=test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:07.906118 24396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:07.912784 24929 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Bootstrap starting.
I20260809 05:21:07.916539 24930 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Bootstrap starting.
I20260809 05:21:07.918175 24929 tablet_bootstrap.cc:654] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:07.919700 24929 log.cc:824] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:07.921859 24930 tablet_bootstrap.cc:654] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:07.923457 24929 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: No bootstrap required, opened a new log
I20260809 05:21:07.923929 24929 ts_tablet_manager.cc:1397] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260809 05:21:07.923995 24933 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Bootstrap starting.
I20260809 05:21:07.924500 24930 log.cc:824] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:07.929605 24933 tablet_bootstrap.cc:654] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:07.932063 24933 log.cc:824] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:07.932649 24930 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: No bootstrap required, opened a new log
I20260809 05:21:07.933197 24930 ts_tablet_manager.cc:1397] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20260809 05:21:07.936275 24933 tablet_bootstrap.cc:492] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: No bootstrap required, opened a new log
I20260809 05:21:07.936789 24933 ts_tablet_manager.cc:1397] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260809 05:21:07.937638 24929 raft_consensus.cc:348] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.938040 24929 raft_consensus.cc:374] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:07.938201 24929 raft_consensus.cc:729] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984d7ddadbca48878b5bc6495277635b, State: Initialized, Role: FOLLOWER
I20260809 05:21:07.938876 24929 consensus_queue.cc:260] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.940210 24905 heartbeater.cc:503] Master 127.22.1.126:35489 was elected leader, sending a full tablet report...
I20260809 05:21:07.942270 24929 ts_tablet_manager.cc:1428] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260809 05:21:07.947958 24930 raft_consensus.cc:348] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.948662 24930 raft_consensus.cc:374] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:07.948860 24930 raft_consensus.cc:729] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8222712b24eb48548de5cee900798905, State: Initialized, Role: FOLLOWER
I20260809 05:21:07.949714 24930 consensus_queue.cc:260] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.950255 24933 raft_consensus.cc:348] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.950857 24933 raft_consensus.cc:374] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:07.951043 24933 raft_consensus.cc:729] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5473333fe553408fba467a1cbeb66365, State: Initialized, Role: FOLLOWER
I20260809 05:21:07.951155 24697 heartbeater.cc:503] Master 127.22.1.126:35489 was elected leader, sending a full tablet report...
I20260809 05:21:07.951814 24933 consensus_queue.cc:260] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:07.952159 24930 ts_tablet_manager.cc:1428] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Time spent starting tablet: real 0.019s user 0.015s sys 0.003s
I20260809 05:21:07.954254 24933 ts_tablet_manager.cc:1428] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
W20260809 05:21:07.992197 24498 tablet.cc:2369] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:08.026372 24700 tablet.cc:2369] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:08.046289 24906 tablet.cc:2369] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:08.320370 24939 raft_consensus.cc:482] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:08.320693 24939 raft_consensus.cc:504] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:08.323221 24939 leader_election.cc:288] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633), 8222712b24eb48548de5cee900798905 (127.22.1.66:42163)
I20260809 05:21:08.333302 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.323817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56947 (local address 127.22.1.67:33633)
0809 05:21:08.324072 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.324078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.325892 (+ 1814us) server_negotiation.cc:407] Connection header received
0809 05:21:08.326095 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.326099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.326182 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.326283 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:08.327091 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.327918 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.330619 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.331995 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.332110 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.332231 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.332742 (+ 511us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.332892 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260809 05:21:08.333734 24926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.323796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:35793 (local address 127.22.1.66:42163)
0809 05:21:08.324053 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.324060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.325629 (+ 1569us) server_negotiation.cc:407] Connection header received
0809 05:21:08.325911 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.325916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.325998 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.326100 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:08.326941 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.328074 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.331051 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.332233 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.332405 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.332540 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.333186 (+ 646us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.333327 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:21:08.333774 24942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.323937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33633 (local address 127.22.1.65:56947)
0809 05:21:08.325775 (+ 1838us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.325792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.325960 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.326324 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.326327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.326368 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:08.326916 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.326930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.328073 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.328079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.330421 (+ 2342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.330435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.332335 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.332344 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.332562 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.332598 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.332680 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.332784 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1658,"mutex_wait_us":3,"thread_start_us":1266,"threads_started":1}
I20260809 05:21:08.334216 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.323548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42163 (local address 127.22.1.65:35793)
0809 05:21:08.325472 (+ 1924us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.325490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.325706 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.326126 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.326131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.326155 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:08.326775 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.326787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.328217 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.328221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.330855 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.330872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.332680 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.332689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.332941 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.332978 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.333099 (+ 121us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.333262 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"mutex_wait_us":42,"thread_start_us":1102,"threads_started":1}
I20260809 05:21:08.334757 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1bb3ad79a41a8a7e34cf9fcc7419b" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b" is_pre_election: true
I20260809 05:21:08.335223 24833 raft_consensus.cc:2454] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 0.
I20260809 05:21:08.335584 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1bb3ad79a41a8a7e34cf9fcc7419b" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8222712b24eb48548de5cee900798905" is_pre_election: true
I20260809 05:21:08.336032 24319 leader_election.cc:302] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5473333fe553408fba467a1cbeb66365, 984d7ddadbca48878b5bc6495277635b; no voters:
I20260809 05:21:08.336338 24632 raft_consensus.cc:2454] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 0.
I20260809 05:21:08.336645 24939 raft_consensus.cc:2781] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:08.336848 24939 raft_consensus.cc:482] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:08.337007 24939 raft_consensus.cc:3037] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.340519 24939 raft_consensus.cc:504] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:08.341359 24939 leader_election.cc:288] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 election: Requested vote from peers 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633), 8222712b24eb48548de5cee900798905 (127.22.1.66:42163)
I20260809 05:21:08.341950 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1bb3ad79a41a8a7e34cf9fcc7419b" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b"
I20260809 05:21:08.341988 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd1bb3ad79a41a8a7e34cf9fcc7419b" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8222712b24eb48548de5cee900798905"
I20260809 05:21:08.342190 24632 raft_consensus.cc:3037] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.342229 24833 raft_consensus.cc:3037] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.344985 24833 raft_consensus.cc:2454] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 1.
I20260809 05:21:08.345544 24319 leader_election.cc:302] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5473333fe553408fba467a1cbeb66365, 984d7ddadbca48878b5bc6495277635b; no voters:
I20260809 05:21:08.345881 24939 raft_consensus.cc:2781] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:08.345919 24632 raft_consensus.cc:2454] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 1.
I20260809 05:21:08.346545 24939 raft_consensus.cc:686] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 1 LEADER]: Becoming Leader. State: Replica: 5473333fe553408fba467a1cbeb66365, State: Running, Role: LEADER
I20260809 05:21:08.347085 24939 consensus_queue.cc:237] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } }
I20260809 05:21:08.353307 24179 catalog_manager.cc:5495] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5473333fe553408fba467a1cbeb66365 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:08.486699 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.478475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.22.1.65:41577)
0809 05:21:08.479001 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.479009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.479308 (+ 299us) server_negotiation.cc:407] Connection header received
0809 05:21:08.479642 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.479646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.479730 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.479881 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:08.480497 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.481552 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.483444 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.484231 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.484360 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.485027 (+ 667us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.485847 (+ 820us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.485960 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.486099 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.486177 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":269,"threads_started":1}
I20260809 05:21:08.497052 24632 raft_consensus.cc:1261] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Refusing update from remote peer 5473333fe553408fba467a1cbeb66365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:08.497957 24833 raft_consensus.cc:1261] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Refusing update from remote peer 5473333fe553408fba467a1cbeb66365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:08.498409 24944 consensus_queue.cc:1035] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:08.498920 24939 consensus_queue.cc:1035] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:08.540612 24178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:08.541021 24178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:08.546017 24178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Sending DeleteTablet for 3 replicas of tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b
I20260809 05:21:08.547778 24396 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:08 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41970
I20260809 05:21:08.547899 24799 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:08 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:46824
I20260809 05:21:08.549124 24597 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:08 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:42178
I20260809 05:21:08.550100 24970 tablet_replica.cc:330] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: stopping tablet replica
I20260809 05:21:08.550961 24970 raft_consensus.cc:2229] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:08.552229 24970 raft_consensus.cc:2258] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:08.552771 24178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:08.553704 24178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:08.555606 24970 ts_tablet_manager.cc:1905] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:08.558023 24969 tablet_replica.cc:330] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: stopping tablet replica
I20260809 05:21:08.558274 24972 tablet_replica.cc:330] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: stopping tablet replica
I20260809 05:21:08.558648 24969 raft_consensus.cc:2229] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:08.558815 24972 raft_consensus.cc:2229] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:08.559299 24969 raft_consensus.cc:2258] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:08.559513 24972 raft_consensus.cc:2258] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:08.561421 24972 ts_tablet_manager.cc:1905] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:08.561419 24969 ts_tablet_manager.cc:1905] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:08.562438 24597 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd95a9e40a844adaddf586a9f56135f (DEFAULT_TABLE table=test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:08.564042 24396 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd95a9e40a844adaddf586a9f56135f (DEFAULT_TABLE table=test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:08.564638 24396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd95a9e40a844adaddf586a9f56135f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.565553 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd95a9e40a844adaddf586a9f56135f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.566164 24799 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd95a9e40a844adaddf586a9f56135f (DEFAULT_TABLE table=test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:08.566992 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd95a9e40a844adaddf586a9f56135f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.578076 24973 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Bootstrap starting.
I20260809 05:21:08.578020 24969 ts_tablet_manager.cc:1918] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:08.579315 24969 log.cc:1196] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/ecd1bb3ad79a41a8a7e34cf9fcc7419b
I20260809 05:21:08.580021 24975 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Bootstrap starting.
I20260809 05:21:08.581728 24973 tablet_bootstrap.cc:654] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.583021 24972 ts_tablet_manager.cc:1918] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:08.583312 24972 log.cc:1196] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/ecd1bb3ad79a41a8a7e34cf9fcc7419b
I20260809 05:21:08.583647 24969 ts_tablet_manager.cc:1939] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 984d7ddadbca48878b5bc6495277635b: Deleting consensus metadata
I20260809 05:21:08.583709 24976 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Bootstrap starting.
I20260809 05:21:08.583981 24972 ts_tablet_manager.cc:1939] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 8222712b24eb48548de5cee900798905: Deleting consensus metadata
I20260809 05:21:08.586066 24155 catalog_manager.cc:4841] TS 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633): tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (table test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]) successfully deleted
I20260809 05:21:08.586532 24153 catalog_manager.cc:4841] TS 8222712b24eb48548de5cee900798905 (127.22.1.66:42163): tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (table test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]) successfully deleted
I20260809 05:21:08.588105 24975 tablet_bootstrap.cc:654] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.588271 24976 tablet_bootstrap.cc:654] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.594151 24975 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: No bootstrap required, opened a new log
I20260809 05:21:08.594410 24975 ts_tablet_manager.cc:1397] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260809 05:21:08.594939 24970 ts_tablet_manager.cc:1918] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:08.595335 24970 log.cc:1196] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/ecd1bb3ad79a41a8a7e34cf9fcc7419b
I20260809 05:21:08.595952 24975 raft_consensus.cc:348] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.596360 24975 raft_consensus.cc:374] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.596578 24970 ts_tablet_manager.cc:1939] T ecd1bb3ad79a41a8a7e34cf9fcc7419b P 5473333fe553408fba467a1cbeb66365: Deleting consensus metadata
I20260809 05:21:08.596604 24975 raft_consensus.cc:729] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8222712b24eb48548de5cee900798905, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.596967 24975 consensus_queue.cc:260] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.599382 24155 catalog_manager.cc:4841] TS 5473333fe553408fba467a1cbeb66365 (127.22.1.65:41577): tablet ecd1bb3ad79a41a8a7e34cf9fcc7419b (table test-table [id=67626f6af4544abf8a2145dbd7a6ebf6]) successfully deleted
I20260809 05:21:08.600183 24975 ts_tablet_manager.cc:1428] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260809 05:21:08.602066 24976 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: No bootstrap required, opened a new log
I20260809 05:21:08.602357 24976 ts_tablet_manager.cc:1397] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20260809 05:21:08.603609 24976 raft_consensus.cc:348] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.603945 24976 raft_consensus.cc:374] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.604166 24976 raft_consensus.cc:729] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5473333fe553408fba467a1cbeb66365, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.604425 24973 tablet_bootstrap.cc:492] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: No bootstrap required, opened a new log
I20260809 05:21:08.604650 24973 ts_tablet_manager.cc:1397] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.017s
I20260809 05:21:08.604517 24976 consensus_queue.cc:260] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.605619 24976 ts_tablet_manager.cc:1428] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:08.605774 24973 raft_consensus.cc:348] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.606155 24973 raft_consensus.cc:374] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.606398 24973 raft_consensus.cc:729] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984d7ddadbca48878b5bc6495277635b, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.606755 24973 consensus_queue.cc:260] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.607963 24973 ts_tablet_manager.cc:1428] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 05:21:08.700621 24955 raft_consensus.cc:482] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:08.701061 24955 raft_consensus.cc:504] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.703079 24955 leader_election.cc:288] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5473333fe553408fba467a1cbeb66365 (127.22.1.65:41577), 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633)
I20260809 05:21:08.713732 24922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.705085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:51057 (local address 127.22.1.67:33633)
0809 05:21:08.705295 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.705301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.706198 (+ 897us) server_negotiation.cc:407] Connection header received
0809 05:21:08.706401 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.706407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.706507 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.706619 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:08.707434 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.708607 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.710948 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.712574 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.712717 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.712822 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.713213 (+ 391us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.713349 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260809 05:21:08.714293 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.704787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33633 (local address 127.22.1.66:51057)
0809 05:21:08.706062 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.706082 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.706251 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.706663 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.706668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.706695 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:08.707224 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.707237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.708741 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.708745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.710759 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.710773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.712796 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.712803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.712995 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.713034 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.713143 (+ 109us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.713317 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":1,"thread_start_us":687,"threads_started":1}
I20260809 05:21:08.714680 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.703819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:34613 (local address 127.22.1.65:41577)
0809 05:21:08.704045 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.704051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.704824 (+ 773us) server_negotiation.cc:407] Connection header received
0809 05:21:08.705168 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.705176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.705266 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.705331 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:08.706895 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.708256 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.711125 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.713118 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.713274 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.713388 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.714208 (+ 820us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.714336 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:21:08.715234 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd95a9e40a844adaddf586a9f56135f" candidate_uuid: "8222712b24eb48548de5cee900798905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b" is_pre_election: true
I20260809 05:21:08.715564 24833 raft_consensus.cc:2454] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8222712b24eb48548de5cee900798905 in term 0.
I20260809 05:21:08.715843 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.703625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41577 (local address 127.22.1.66:34613)
0809 05:21:08.704723 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.704742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.704946 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.705565 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.705570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.705599 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:08.706686 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.706704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.708419 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.708425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.710956 (+ 2531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.710974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.713717 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.713727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.714006 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.714040 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.714878 (+ 838us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.715021 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":64,"thread_start_us":667,"threads_started":1}
I20260809 05:21:08.716219 24533 leader_election.cc:302] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8222712b24eb48548de5cee900798905, 984d7ddadbca48878b5bc6495277635b; no voters:
I20260809 05:21:08.716913 24955 raft_consensus.cc:2781] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:08.717211 24955 raft_consensus.cc:482] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:08.717406 24955 raft_consensus.cc:3037] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.718060 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd95a9e40a844adaddf586a9f56135f" candidate_uuid: "8222712b24eb48548de5cee900798905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5473333fe553408fba467a1cbeb66365" is_pre_election: true
I20260809 05:21:08.718904 24423 raft_consensus.cc:2454] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8222712b24eb48548de5cee900798905 in term 0.
I20260809 05:21:08.720463 24955 raft_consensus.cc:504] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.721367 24955 leader_election.cc:288] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [CANDIDATE]: Term 1 election: Requested vote from peers 5473333fe553408fba467a1cbeb66365 (127.22.1.65:41577), 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633)
I20260809 05:21:08.722057 24423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd95a9e40a844adaddf586a9f56135f" candidate_uuid: "8222712b24eb48548de5cee900798905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5473333fe553408fba467a1cbeb66365"
I20260809 05:21:08.721994 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd95a9e40a844adaddf586a9f56135f" candidate_uuid: "8222712b24eb48548de5cee900798905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b"
I20260809 05:21:08.722330 24423 raft_consensus.cc:3037] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.722381 24833 raft_consensus.cc:3037] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.726454 24833 raft_consensus.cc:2454] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8222712b24eb48548de5cee900798905 in term 1.
I20260809 05:21:08.726991 24533 leader_election.cc:302] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8222712b24eb48548de5cee900798905, 984d7ddadbca48878b5bc6495277635b; no voters:
I20260809 05:21:08.727412 24955 raft_consensus.cc:2781] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:08.727561 24423 raft_consensus.cc:2454] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8222712b24eb48548de5cee900798905 in term 1.
I20260809 05:21:08.728433 24955 raft_consensus.cc:686] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 1 LEADER]: Becoming Leader. State: Replica: 8222712b24eb48548de5cee900798905, State: Running, Role: LEADER
I20260809 05:21:08.728997 24955 consensus_queue.cc:237] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.733798 24178 catalog_manager.cc:5495] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8222712b24eb48548de5cee900798905 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "8222712b24eb48548de5cee900798905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } health_report { overall_health: UNKNOWN } } }
Aug 09 05:21:08 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252865, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:08.836637 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.801212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35489 (local address 127.0.0.1:60712)
0809 05:21:08.814268 (+ 13056us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.814292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.814494 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.815227 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.815233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.816280 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:08.816978 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.816993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.819067 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.819073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.821201 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.821218 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.822596 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.822607 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.823008 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.824014 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:08.824045 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:08.827998 (+ 3953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:08.833537 (+ 5539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:08.833550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:08.833559 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:08.834314 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:08.834844 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:08.834849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:08.834853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:08.834997 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:08.835590 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:08.835601 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:08.835775 (+ 174us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.835914 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.836198 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12835,"thread_start_us":12490,"threads_started":1}
I20260809 05:21:08.837057 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.800979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.22.1.126:35489)
0809 05:21:08.814800 (+ 13821us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.814807 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.814833 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:21:08.814895 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.814900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.814995 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.815123 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:08.817172 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.818592 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.821348 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.822340 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.822465 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.828278 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:08.828286 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:08.828293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:08.828334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:08.833392 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:08.834453 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:08.834459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:08.834462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:08.834738 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:08.835114 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:08.835117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:08.835119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:08.835454 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:08.835647 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.836046 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.836394 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":13690,"thread_start_us":13495,"threads_started":1}
I20260809 05:21:08.843533 24178 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:60712:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 05:21:08.844875 24178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:08.852737 24396 tablet_service.cc:1467] Processing CreateTablet for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c9c64abc5104c3a859c1a49da3d0232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:08.853024 24597 tablet_service.cc:1467] Processing CreateTablet for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c9c64abc5104c3a859c1a49da3d0232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:08.853385 24396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.853721 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.856196 24799 tablet_service.cc:1467] Processing CreateTablet for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9c9c64abc5104c3a859c1a49da3d0232]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:08.856925 24799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b00c16ac4ea94188ac4c4c9e59b4c1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:08.863171 24975 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: Bootstrap starting.
I20260809 05:21:08.864988 24973 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: Bootstrap starting.
I20260809 05:21:08.865679 24975 tablet_bootstrap.cc:654] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.865937 24976 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: Bootstrap starting.
I20260809 05:21:08.867856 24973 tablet_bootstrap.cc:654] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.868747 24976 tablet_bootstrap.cc:654] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:08.871871 24976 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: No bootstrap required, opened a new log
I20260809 05:21:08.872129 24976 ts_tablet_manager.cc:1397] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260809 05:21:08.873661 24976 raft_consensus.cc:348] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.874099 24976 raft_consensus.cc:374] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.874245 24976 raft_consensus.cc:729] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5473333fe553408fba467a1cbeb66365, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.874562 24976 consensus_queue.cc:260] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.875070 24939 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.875468 24939 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.876029 24976 ts_tablet_manager.cc:1428] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260809 05:21:08.878201 24975 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: No bootstrap required, opened a new log
I20260809 05:21:08.878522 24975 ts_tablet_manager.cc:1397] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260809 05:21:08.879665 24973 tablet_bootstrap.cc:492] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: No bootstrap required, opened a new log
I20260809 05:21:08.879923 24973 ts_tablet_manager.cc:1397] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20260809 05:21:08.880854 24973 raft_consensus.cc:348] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.880877 24975 raft_consensus.cc:348] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.881328 24973 raft_consensus.cc:374] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.881358 24975 raft_consensus.cc:374] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:08.881505 24973 raft_consensus.cc:729] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 984d7ddadbca48878b5bc6495277635b, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.881562 24975 raft_consensus.cc:729] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8222712b24eb48548de5cee900798905, State: Initialized, Role: FOLLOWER
I20260809 05:21:08.881865 24973 consensus_queue.cc:260] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.881889 24975 consensus_queue.cc:260] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.882431 24955 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.882514 24956 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.882848 24955 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.883011 24956 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.883392 24975 ts_tablet_manager.cc:1428] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260809 05:21:08.883795 24939 raft_consensus.cc:482] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:08.884035 24939 raft_consensus.cc:504] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.884972 24939 leader_election.cc:288] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8222712b24eb48548de5cee900798905 (127.22.1.66:42163), 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633)
I20260809 05:21:08.885984 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c16ac4ea94188ac4c4c9e59b4c1b2" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8222712b24eb48548de5cee900798905" is_pre_election: true
I20260809 05:21:08.886304 24632 raft_consensus.cc:2454] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 0.
I20260809 05:21:08.887331 24317 leader_election.cc:302] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5473333fe553408fba467a1cbeb66365, 8222712b24eb48548de5cee900798905; no voters:
I20260809 05:21:08.887344 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c16ac4ea94188ac4c4c9e59b4c1b2" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b" is_pre_election: true
I20260809 05:21:08.887645 24939 raft_consensus.cc:2781] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:08.887665 24833 raft_consensus.cc:2454] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 0.
I20260809 05:21:08.887821 24939 raft_consensus.cc:482] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:08.888024 24939 raft_consensus.cc:3037] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.886250 24973 ts_tablet_manager.cc:1428] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260809 05:21:08.891533 24939 raft_consensus.cc:504] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.892277 24939 leader_election.cc:288] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 election: Requested vote from peers 8222712b24eb48548de5cee900798905 (127.22.1.66:42163), 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633)
I20260809 05:21:08.892789 24632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c16ac4ea94188ac4c4c9e59b4c1b2" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8222712b24eb48548de5cee900798905"
I20260809 05:21:08.892846 24833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b00c16ac4ea94188ac4c4c9e59b4c1b2" candidate_uuid: "5473333fe553408fba467a1cbeb66365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "984d7ddadbca48878b5bc6495277635b"
I20260809 05:21:08.893065 24632 raft_consensus.cc:3037] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.893245 24833 raft_consensus.cc:3037] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:08.896957 24833 raft_consensus.cc:2454] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 1.
I20260809 05:21:08.896982 24632 raft_consensus.cc:2454] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5473333fe553408fba467a1cbeb66365 in term 1.
I20260809 05:21:08.897517 24317 leader_election.cc:302] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5473333fe553408fba467a1cbeb66365, 8222712b24eb48548de5cee900798905; no voters:
I20260809 05:21:08.897945 24939 raft_consensus.cc:2781] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:08.898156 24939 raft_consensus.cc:686] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [term 1 LEADER]: Becoming Leader. State: Replica: 5473333fe553408fba467a1cbeb66365, State: Running, Role: LEADER
I20260809 05:21:08.898600 24939 consensus_queue.cc:237] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } }
I20260809 05:21:08.901400 24963 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: TxnStatusTablet state changed. Reason: New leader 5473333fe553408fba467a1cbeb66365. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.902323 24963 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:08.903271 24179 catalog_manager.cc:5495] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5473333fe553408fba467a1cbeb66365 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:08.905260 25003 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 05:21:08.905591 25003 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 05:21:08.933296 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.925937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41577 (local address 127.0.0.1:41982)
0809 05:21:08.926329 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.926347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.926506 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.926984 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.926989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.927001 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:08.927327 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.927339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.928782 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.928788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.930379 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.930392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.931556 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.931564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.931901 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.931968 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.932486 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.932503 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.932600 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.932771 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260809 05:21:08.933347 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.925906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.22.1.65:41577)
0809 05:21:08.926149 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.926155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.926414 (+ 259us) server_negotiation.cc:407] Connection header received
0809 05:21:08.926701 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.926706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.926846 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.926938 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:08.927503 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.928649 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.930519 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.931354 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.931472 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.932157 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.932351 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.932498 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.932657 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.932779 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:21:08.938149 24833 raft_consensus.cc:1261] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Refusing update from remote peer 5473333fe553408fba467a1cbeb66365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:08.938220 24632 raft_consensus.cc:1261] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Refusing update from remote peer 5473333fe553408fba467a1cbeb66365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:08.938861 24963 consensus_queue.cc:1035] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:08.939179 24939 consensus_queue.cc:1035] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:08.946718 24956 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: TxnStatusTablet state changed. Reason: New leader 5473333fe553408fba467a1cbeb66365. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.947118 24956 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.948829 24963 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.950429 24954 mvcc.cc:205] Tried to move back new op lower bound from 7316491751162851328 to 7316491751013036032. Current Snapshot: MvccSnapshot[applied={T|T < 7316491751162851328 or (T in {7316491751162851328})}]
I20260809 05:21:08.950769 24963 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:08.954780 24955 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: TxnStatusTablet state changed. Reason: New leader 5473333fe553408fba467a1cbeb66365. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.955060 24956 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.955284 24956 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 984d7ddadbca48878b5bc6495277635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.955385 24955 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.958042 24963 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.958338 24963 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 5473333fe553408fba467a1cbeb66365: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:08.960721 24955 tablet_replica.cc:439] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5473333fe553408fba467a1cbeb66365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 } } peers { permanent_uuid: "8222712b24eb48548de5cee900798905" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42163 } } peers { permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 } } }
I20260809 05:21:08.961061 24955 tablet_replica.cc:442] T b00c16ac4ea94188ac4c4c9e59b4c1b2 P 8222712b24eb48548de5cee900798905: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:08.969584 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.961866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.22.1.66:42163)
0809 05:21:08.962789 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.962794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.962816 (+ 22us) server_negotiation.cc:407] Connection header received
0809 05:21:08.962905 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.962909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.963061 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.963201 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:08.963654 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.964575 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.966580 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.967192 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.967285 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.967890 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.968561 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.968666 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.968904 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.968975 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":708,"threads_started":1}
I20260809 05:21:08.993079 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.983248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:46243 (local address 127.22.1.65:41577)
0809 05:21:08.983462 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:08.983468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:08.984601 (+ 1133us) server_negotiation.cc:407] Connection header received
0809 05:21:08.984849 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:08.984853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:08.984918 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:08.984994 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:08.986443 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.987770 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.990492 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.991377 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.991476 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.992248 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.992386 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.992456 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:08.992707 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 05:21:08.992777 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:21:08.993559 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:08.983226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41577 (local address 127.22.1.66:46243)
0809 05:21:08.984481 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:08.984505 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:08.984707 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:08.985052 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:08.985057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:08.985072 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:08.986191 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.986208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.987978 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.987984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.990320 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:08.990338 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:08.991637 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:08.991647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:08.991908 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:08.992006 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:08.992507 (+ 501us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:08.992560 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 05:21:08.992660 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 05:21:08.992810 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":619,"threads_started":1}
I20260809 05:21:09.006314 24833 raft_consensus.cc:1261] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Refusing update from remote peer 8222712b24eb48548de5cee900798905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:09.006610 24423 raft_consensus.cc:1261] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Refusing update from remote peer 8222712b24eb48548de5cee900798905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:09.007301 24985 consensus_queue.cc:1035] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "984d7ddadbca48878b5bc6495277635b" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:09.007963 24955 consensus_queue.cc:1035] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5473333fe553408fba467a1cbeb66365" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:09.015933 24958 mvcc.cc:205] Tried to move back new op lower bound from 7316491751441530880 to 7316491750319587328. Current Snapshot: MvccSnapshot[applied={T|T < 7316491751441530880}]
I20260809 05:21:09.018069 24957 mvcc.cc:205] Tried to move back new op lower bound from 7316491751441530880 to 7316491750319587328. Current Snapshot: MvccSnapshot[applied={T|T < 7316491751441530880}]
I20260809 05:21:09.067512 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.055796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:33225 (local address 127.22.1.66:42163)
0809 05:21:09.056000 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.056006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.059124 (+ 3118us) server_negotiation.cc:407] Connection header received
0809 05:21:09.059304 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.059309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.059405 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.059527 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:09.060328 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.061717 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.064719 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.065576 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.065704 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.066565 (+ 861us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:09.066764 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:09.066821 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.067003 (+ 182us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.067085 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 05:21:09.067718 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.055888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42163 (local address 127.22.1.65:33225)
0809 05:21:09.058997 (+ 3109us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:09.059013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:09.059163 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:09.059535 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:09.059539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:09.059549 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:09.060124 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:09.060135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.061931 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.061937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:09.064552 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:09.064570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.065943 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.065961 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:09.066289 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.066365 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:09.066885 (+ 520us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:09.066892 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 05:21:09.066971 (+ 79us) client_negotiation.cc:240] Negotiation successful
0809 05:21:09.067115 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2808,"thread_start_us":2537,"threads_started":1}
I20260809 05:21:09.126231 24178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:09.126677 24178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60702:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:09.130988 24178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9482b5c8f97d4a1d90e14d0e6f820f53: Sending DeleteTablet for 3 replicas of tablet 7dd95a9e40a844adaddf586a9f56135f
I20260809 05:21:09.132177 24597 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd95a9e40a844adaddf586a9f56135f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:42178
I20260809 05:21:09.132820 24396 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd95a9e40a844adaddf586a9f56135f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41970
I20260809 05:21:09.133363 25032 tablet_replica.cc:330] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: stopping tablet replica
I20260809 05:21:09.133726 25032 raft_consensus.cc:2229] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:09.134352 24799 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd95a9e40a844adaddf586a9f56135f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:46824
I20260809 05:21:09.134794 25032 raft_consensus.cc:2258] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:09.136212 25032 ts_tablet_manager.cc:1905] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:09.139740 25034 tablet_replica.cc:330] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: stopping tablet replica
I20260809 05:21:09.140028 25034 raft_consensus.cc:2229] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:09.140833 25034 raft_consensus.cc:2258] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:09.142114 25034 ts_tablet_manager.cc:1905] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:09.150800 25034 ts_tablet_manager.cc:1918] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:09.151105 25034 log.cc:1196] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/7dd95a9e40a844adaddf586a9f56135f
I20260809 05:21:09.152045 25034 ts_tablet_manager.cc:1939] T 7dd95a9e40a844adaddf586a9f56135f P 984d7ddadbca48878b5bc6495277635b: Deleting consensus metadata
I20260809 05:21:09.153399 24155 catalog_manager.cc:4841] TS 984d7ddadbca48878b5bc6495277635b (127.22.1.67:33633): tablet 7dd95a9e40a844adaddf586a9f56135f (table test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]) successfully deleted
I20260809 05:21:09.153861 25032 ts_tablet_manager.cc:1918] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:09.154204 25032 log.cc:1196] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/7dd95a9e40a844adaddf586a9f56135f
I20260809 05:21:09.155087 25032 ts_tablet_manager.cc:1939] T 7dd95a9e40a844adaddf586a9f56135f P 8222712b24eb48548de5cee900798905: Deleting consensus metadata
I20260809 05:21:09.156463 24153 catalog_manager.cc:4841] TS 8222712b24eb48548de5cee900798905 (127.22.1.66:42163): tablet 7dd95a9e40a844adaddf586a9f56135f (table test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]) successfully deleted
I20260809 05:21:09.158869 25033 tablet_replica.cc:330] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: stopping tablet replica
I20260809 05:21:09.159212 25033 raft_consensus.cc:2229] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:09.159761 25033 raft_consensus.cc:2258] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:09.161392 25033 ts_tablet_manager.cc:1905] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:09.172207 25033 ts_tablet_manager.cc:1918] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:09.172607 25033 log.cc:1196] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/7dd95a9e40a844adaddf586a9f56135f
I20260809 05:21:09.173756 25033 ts_tablet_manager.cc:1939] T 7dd95a9e40a844adaddf586a9f56135f P 5473333fe553408fba467a1cbeb66365: Deleting consensus metadata
I20260809 05:21:09.175262 24155 catalog_manager.cc:4841] TS 5473333fe553408fba467a1cbeb66365 (127.22.1.65:41577): tablet 7dd95a9e40a844adaddf586a9f56135f (table test-table [id=f02dbe1988ad4091b7d2ce57b1d59783]) successfully deleted
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:09.193461 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.159324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.22.1.65:41577)
0809 05:21:09.159528 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.159535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.164358 (+ 4823us) server_negotiation.cc:407] Connection header received
0809 05:21:09.164962 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.164968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.165071 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.165210 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:09.168693 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.169884 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.172757 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.173867 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.173978 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.182235 (+ 8257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:09.182264 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:09.182294 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:09.182359 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:09.188467 (+ 6108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.189871 (+ 1404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.189883 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.189887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.190088 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.190648 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.190654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.190657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.191359 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:09.191815 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.192106 (+ 291us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.192387 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260809 05:21:09.194607 24489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41990
I20260809 05:21:09.223098 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.199146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.22.1.126:35489)
0809 05:21:09.199339 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.199345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.201801 (+ 2456us) server_negotiation.cc:407] Connection header received
0809 05:21:09.202031 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.202037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.202123 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.202236 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:09.205510 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.206993 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.209051 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.210112 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.210259 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.214246 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:09.214254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:09.214260 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:09.214310 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:09.218972 (+ 4662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.219973 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.219980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.219984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.220152 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.220631 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.220637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.220641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.221235 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:09.221671 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.222007 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.222394 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 05:21:09.224013 24178 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60714
I20260809 05:21:09.228570 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24295
I20260809 05:21:09.248572 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.229607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.22.1.65:41577)
0809 05:21:09.229836 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.229845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.230732 (+ 887us) server_negotiation.cc:407] Connection header received
0809 05:21:09.230918 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.230945 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.231031 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.231135 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:09.232787 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.233993 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.235828 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.236467 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.236620 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.239745 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:09.239753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:09.239758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:09.239808 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:09.245036 (+ 5228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.246157 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.246163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.246167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.246341 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.246821 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.246827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.246830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.247279 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:09.247598 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.247905 (+ 307us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.248168 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260809 05:21:09.249367 24489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41996
W20260809 05:21:09.250061 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:09.250408 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24295
I20260809 05:21:09.277661 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24502
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:09.293423 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.278443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.22.1.66:42163)
0809 05:21:09.278652 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.278657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.278753 (+ 96us) server_negotiation.cc:407] Connection header received
0809 05:21:09.278963 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.278968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.279047 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.279147 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:09.280396 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.281402 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.282660 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.283117 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.283232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.287332 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:09.287346 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:09.287362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:09.287416 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:09.290525 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.291214 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.291226 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.291228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.291382 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.291772 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.291776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.291778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.292171 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:09.292547 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.292806 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.293052 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260809 05:21:09.294124 24692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42196
W20260809 05:21:09.294773 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:09.294921 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24502
I20260809 05:21:09.318641 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24704
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[24109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252867, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:09.334934 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:09.319472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.22.1.67:33633)
0809 05:21:09.320099 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:09.320104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:09.320120 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:21:09.320176 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:09.320180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:09.320241 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:09.320331 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:09.321688 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.322461 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.323857 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:09.324300 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:09.324408 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:09.329052 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:09.329070 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:09.329089 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:09.329129 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:09.332047 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.332845 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.332853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.332855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.333015 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:09.333476 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:09.333480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:09.333482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:09.333896 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:09.334132 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:09.334470 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 05:21:09.334665 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":357,"threads_started":1}
W20260809 05:21:09.335640 24898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46838
W20260809 05:21:09.336372 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24704: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:09.336570 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24704
I20260809 05:21:09.356868 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24133
I20260809 05:21:09.358191 24236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:09.471992 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24133
2026-08-09T05:21:09Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5210 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[25047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[25047](info): set up 2 sockets
Aug 09 05:21:09 dist-test-slave-rmhg krb5kdc[25047](info): commencing operation
krb5kdc: starting...
W20260809 05:21:10.682468 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.162s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:10 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:10.889618 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41293
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:11.229065 25062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:11.229424 25062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:11.229568 25062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:11.236905 25062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:11.237058 25062 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 05:21:11.237120 25062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:11.237223 25062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:11.237289 25062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:11.237375 25062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:11.247895 25062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41293
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:11.249387 25062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:11.250808 25062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:11.263304 25071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:11.263355 25075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:11.263298 25072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:11.264333 25062 server_base.cc:1034] running on GCE node
I20260809 05:21:11.265282 25062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:11.266829 25062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:11.268060 25062 hybrid_clock.cc:648] HybridClock initialized: now 1786252871267986 us; error 50 us; skew 500 ppm
I20260809 05:21:11.268402 25062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:11 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:11.271842 25062 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:11.273434 25062 webserver.cc:460] Webserver started at http://127.22.1.126:33179/ using document root <none> and password file <none>
I20260809 05:21:11.274163 25062 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:11.274261 25062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:11.274600 25062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:11.277834 25062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "d80f4375944d40d2868ff543d9b7f66d"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "69582719b13c818c4b6926a3599d7402"
server_key_iv: "f7cfd1f2b2d1c53d9cbd5db372fef076"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.278692 25062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "d80f4375944d40d2868ff543d9b7f66d"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "69582719b13c818c4b6926a3599d7402"
server_key_iv: "f7cfd1f2b2d1c53d9cbd5db372fef076"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.284540 25062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:21:11.288574 25085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:11.290621 25062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:21:11.290827 25062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "d80f4375944d40d2868ff543d9b7f66d"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "69582719b13c818c4b6926a3599d7402"
server_key_iv: "f7cfd1f2b2d1c53d9cbd5db372fef076"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.291105 25062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:11.312072 25062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:11.316551 25062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:11.316833 25062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:11.338169 25062 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41293
I20260809 05:21:11.338181 25183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41293 every 8 connection(s)
I20260809 05:21:11.339625 25062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:11.343114 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 25062
I20260809 05:21:11.343379 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:11.344054 25184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:11.344256 22533 external_mini_cluster.cc:1428] Setting key 43720d339b16aba661430c8973b75e28
I20260809 05:21:11.359352 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Bootstrap starting.
Aug 09 05:21:11 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:11.363408 25184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:11.364904 25184 log.cc:824] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:11.368721 25184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: No bootstrap required, opened a new log
I20260809 05:21:11.370236 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:11.346567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.22.1.126:41293)
0809 05:21:11.347399 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:11.347413 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:11.347494 (+ 81us) server_negotiation.cc:407] Connection header received
0809 05:21:11.349115 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:11.349143 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:11.349673 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:11.350056 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:11.351976 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.353395 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.355203 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.356002 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.356172 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:11.361050 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:11.361063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:11.361085 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:11.361140 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:11.365202 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.366170 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.366177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.366184 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.366376 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.366807 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.366812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.366815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.367595 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:11.367896 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:11.368303 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 05:21:11.368772 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":380,"threads_started":1}
I20260809 05:21:11.377427 25184 raft_consensus.cc:348] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } }
I20260809 05:21:11.377868 25184 raft_consensus.cc:374] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:11.377985 25184 raft_consensus.cc:729] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80f4375944d40d2868ff543d9b7f66d, State: Initialized, Role: FOLLOWER
I20260809 05:21:11.378602 25184 consensus_queue.cc:260] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } }
I20260809 05:21:11.378831 25184 raft_consensus.cc:388] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:11.378968 25184 raft_consensus.cc:482] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:11.379132 25184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:11.381623 25184 raft_consensus.cc:504] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } }
I20260809 05:21:11.382133 25184 leader_election.cc:302] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80f4375944d40d2868ff543d9b7f66d; no voters:
I20260809 05:21:11.382678 25184 leader_election.cc:288] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:11.382808 25193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:11.383450 25193 raft_consensus.cc:686] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [term 1 LEADER]: Becoming Leader. State: Replica: d80f4375944d40d2868ff543d9b7f66d, State: Running, Role: LEADER
I20260809 05:21:11.383960 25193 consensus_queue.cc:237] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } }
I20260809 05:21:11.384716 25184 sys_catalog.cc:564] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:11.386740 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d80f4375944d40d2868ff543d9b7f66d. Latest consensus state: current_term: 1 leader_uuid: "d80f4375944d40d2868ff543d9b7f66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } } }
I20260809 05:21:11.387194 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:11.387627 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d80f4375944d40d2868ff543d9b7f66d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80f4375944d40d2868ff543d9b7f66d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41293 } } }
I20260809 05:21:11.387902 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:11.388736 25204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:11.393170 25204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:11.402678 25204 catalog_manager.cc:1347] Generated new cluster ID: 28cc4983d8e34d8fa7d51efced174064
I20260809 05:21:11.402808 25204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:11.437829 25204 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:11.439783 25204 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:11.447542 25204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Generated new TSK 0
I20260809 05:21:11.448633 25204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:11.501288 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41293
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:11.837383 25225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:11.837687 25225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:11.837790 25225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:11.844952 25225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:11.845072 25225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:11.845177 25225 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 05:21:11.845249 25225 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:11.845326 25225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:11.855273 25225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:41293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:11.856575 25225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:11.857928 25225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:11.870812 25233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:11.870828 25237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:11.871037 25231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:11.871647 25225 server_base.cc:1034] running on GCE node
I20260809 05:21:11.872295 25225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:11.873291 25225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:11.874536 25225 hybrid_clock.cc:648] HybridClock initialized: now 1786252871874475 us; error 47 us; skew 500 ppm
I20260809 05:21:11.874851 25225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:11 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:11.881021 25225 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:11.882556 25225 webserver.cc:460] Webserver started at http://127.22.1.65:46497/ using document root <none> and password file <none>
I20260809 05:21:11.883190 25225 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:11.883291 25225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:11.883569 25225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:11.886528 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "09299ab69a744b29bbd601a0ab09ee35"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "64ba770330c5ec475959157ce47dab2d"
server_key_iv: "47d135d4868aa5b2855dc251deaa2ed2"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.887203 25225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "09299ab69a744b29bbd601a0ab09ee35"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "64ba770330c5ec475959157ce47dab2d"
server_key_iv: "47d135d4868aa5b2855dc251deaa2ed2"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.892307 25225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:21:11.896129 25247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:11.897661 25225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:11.897807 25225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "09299ab69a744b29bbd601a0ab09ee35"
format_stamp: "Formatted at 2026-08-09 05:21:11 on dist-test-slave-rmhg"
server_key: "64ba770330c5ec475959157ce47dab2d"
server_key_iv: "47d135d4868aa5b2855dc251deaa2ed2"
server_key_version: "encryptionkey@0"
I20260809 05:21:11.897977 25225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:11.926427 25225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:11.930801 25225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:11.931056 25225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:11.933864 25225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:11.933995 25225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:11.934114 25225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:11.934254 25225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 09 05:21:11 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:11.956719 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:11.934393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:44511 (local address 127.22.1.126:41293)
0809 05:21:11.935165 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:11.935170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:11.936164 (+ 994us) server_negotiation.cc:407] Connection header received
0809 05:21:11.938191 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:11.938197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:11.938280 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:11.938374 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:11.940570 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.941712 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.944142 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.944822 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.944958 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:11.949323 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:11.949331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:11.949336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:11.949382 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:11.952731 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.953870 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.953876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.953880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.954065 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.954574 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.954580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.954584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.954962 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:11.955136 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:11.956097 (+ 961us) server_negotiation.cc:299] Negotiation successful
0809 05:21:11.956348 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":571,"threads_started":1}
I20260809 05:21:11.958164 25262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:11.935115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.65:44511)
0809 05:21:11.935984 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:11.936031 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:11.937885 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:11.938648 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:11.938656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:11.939424 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:11.940372 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:11.940392 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.941896 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.941905 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:11.943991 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:11.944005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.945051 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.945056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:11.945290 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:11.946073 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:11.946101 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:11.949029 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:11.952921 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:11.952933 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:11.952956 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:11.953717 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:11.954212 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:11.954216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:11.954218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:11.954440 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:11.955132 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:11.955141 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:11.955634 (+ 493us) client_negotiation.cc:769] Sending connection context
0809 05:21:11.955990 (+ 356us) client_negotiation.cc:240] Negotiation successful
0809 05:21:11.956557 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":384,"threads_started":1}
I20260809 05:21:11.976773 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:11.963886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.65:42853)
0809 05:21:11.964524 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:11.964540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:11.964654 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:11.964964 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:11.964967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:11.965510 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:11.965976 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:11.965989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.966991 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.966997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:11.967998 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:11.968010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.968724 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.968730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:11.968892 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:11.969645 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:11.969668 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:11.970616 (+ 948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:11.974088 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:11.974094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:11.974098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:11.974686 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:11.975079 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:11.975085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:11.975089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:11.975218 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:11.975727 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:11.975735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:11.975936 (+ 201us) client_negotiation.cc:769] Sending connection context
0809 05:21:11.976109 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 05:21:11.976447 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
I20260809 05:21:11.976801 25261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:11.963852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:42853 (local address 127.22.1.126:41293)
0809 05:21:11.964006 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:11.964010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:11.964642 (+ 632us) server_negotiation.cc:407] Connection header received
0809 05:21:11.964749 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:11.964755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:11.964846 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:11.964949 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:11.966137 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.966851 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.968138 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:11.968533 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:11.968641 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:11.970804 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:11.970809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:11.970812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:11.970849 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:11.973975 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.974817 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.974823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.974826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.974965 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:11.975328 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:11.975333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:11.975336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:11.975604 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:11.975732 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:11.976240 (+ 508us) server_negotiation.cc:299] Negotiation successful
0809 05:21:11.976474 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:21:11.985497 25225 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:41361
I20260809 05:21:11.985571 25426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:41361 every 8 connection(s)
I20260809 05:21:11.986821 25225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:11.988989 25427 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41293
I20260809 05:21:11.989270 25427 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:11.989943 25427 heartbeater.cc:511] Master 127.22.1.126:41293 requested a full tablet report, sending...
I20260809 05:21:11.992112 25110 ts_manager.cc:194] Registered new tserver with Master: 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361)
I20260809 05:21:11.992622 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 25225
I20260809 05:21:11.992857 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:11.993686 22533 external_mini_cluster.cc:1428] Setting key 4e905d291aefc66d73733f56ce578107
I20260809 05:21:11.994221 25110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:44511
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:12.038064 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41293
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:12.377767 25434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:12.378062 25434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:12.378199 25434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:12.386708 25434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:12.386831 25434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:12.386976 25434 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 05:21:12.387040 25434 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:12.387147 25434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:12.397486 25434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:41293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:12.398877 25434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:12.400270 25434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:12.414335 25439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:12.414394 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:12.414363 25441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:12.415139 25434 server_base.cc:1034] running on GCE node
I20260809 05:21:12.415809 25434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:12.416864 25434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:12.418087 25434 hybrid_clock.cc:648] HybridClock initialized: now 1786252872418010 us; error 71 us; skew 500 ppm
I20260809 05:21:12.418424 25434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:12 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252872, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:12.421690 25434 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:12.423200 25434 webserver.cc:460] Webserver started at http://127.22.1.66:35001/ using document root <none> and password file <none>
I20260809 05:21:12.423913 25434 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:12.424039 25434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:12.424353 25434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:12.427466 25434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "07fa601ab397415e9397148239fa13f0"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "3b94cc70d1f527bda73eb7b0a46dd659"
server_key_iv: "7e9ea5b68544908355df13fcb27cb5fe"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.428241 25434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "07fa601ab397415e9397148239fa13f0"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "3b94cc70d1f527bda73eb7b0a46dd659"
server_key_iv: "7e9ea5b68544908355df13fcb27cb5fe"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.433725 25434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:12.437587 25454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:12.439083 25434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:12.439257 25434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "07fa601ab397415e9397148239fa13f0"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "3b94cc70d1f527bda73eb7b0a46dd659"
server_key_iv: "7e9ea5b68544908355df13fcb27cb5fe"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.439453 25434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:12.478883 25434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:12.483604 25434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:12.483907 25434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:12.486882 25434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:12.486996 25434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:12.487112 25434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:12.487187 25434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:21:12 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252872, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:12.512848 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:12.487497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:42893 (local address 127.22.1.126:41293)
0809 05:21:12.488109 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:12.488116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:12.489434 (+ 1318us) server_negotiation.cc:407] Connection header received
0809 05:21:12.491558 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:12.491565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:12.491662 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:12.491799 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:12.494231 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.495318 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.498154 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.498740 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.498888 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:12.504731 (+ 5843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:12.504740 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:12.504746 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:12.504795 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:12.508462 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:12.509676 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:12.509682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:12.509685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:12.509857 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:12.510471 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:12.510476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:12.510480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:12.510895 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:12.511052 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:12.512149 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0809 05:21:12.512410 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":324,"threads_started":1}
I20260809 05:21:12.514882 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:12.488305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.66:42893)
0809 05:21:12.489235 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:12.489297 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:12.491261 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:12.492120 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:12.492136 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:12.493009 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:12.493969 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:12.493998 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.495537 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.495543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:12.497965 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:12.497987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.499086 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.499095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:12.499441 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:12.500308 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:12.500354 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:12.504369 (+ 4015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:12.508667 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:12.508682 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:12.508702 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:12.509464 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:12.509999 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:12.510004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:12.510009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:12.510317 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:12.511053 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:12.511061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:12.511611 (+ 550us) client_negotiation.cc:769] Sending connection context
0809 05:21:12.512019 (+ 408us) client_negotiation.cc:240] Negotiation successful
0809 05:21:12.512710 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":381,"threads_started":1}
I20260809 05:21:12.526939 25434 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:42135
I20260809 05:21:12.527017 25637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:42135 every 8 connection(s)
I20260809 05:21:12.528354 25434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:12.528756 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 25434
I20260809 05:21:12.528960 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:12.529548 22533 external_mini_cluster.cc:1428] Setting key 11bee65afbdf0d978d149d9a8e47fc73
I20260809 05:21:12.531055 25638 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41293
I20260809 05:21:12.531402 25638 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:12.532464 25638 heartbeater.cc:511] Master 127.22.1.126:41293 requested a full tablet report, sending...
I20260809 05:21:12.534379 25110 ts_manager.cc:194] Registered new tserver with Master: 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:12.535679 25110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:42893
I20260809 05:21:12.541239 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:12.525713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:41543 (local address 127.22.1.126:41293)
0809 05:21:12.525929 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:12.525936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:12.526693 (+ 757us) server_negotiation.cc:407] Connection header received
0809 05:21:12.526912 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:12.526919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:12.527017 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:12.527129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:12.528269 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.529406 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.531374 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.531946 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.532061 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:12.534747 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:12.534755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:12.534760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:12.534801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:12.538128 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:12.538968 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:12.538973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:12.538977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:12.539132 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:12.539553 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:12.539557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:12.539560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:12.539969 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:12.540095 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:12.540648 (+ 553us) server_negotiation.cc:299] Negotiation successful
0809 05:21:12.540869 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260809 05:21:12.541314 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:12.525736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.66:41543)
0809 05:21:12.526539 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:12.526569 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:12.526761 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:12.527326 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:12.527330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:12.527806 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:12.528104 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:12.528113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.529618 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.529624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:12.531204 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:12.531220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:12.532389 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:12.532396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:12.532597 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:12.533424 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:12.533450 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:12.534499 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:12.538309 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:12.538314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:12.538317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:12.538831 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:12.539273 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:12.539276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:12.539279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:12.539429 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:12.540188 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:12.540192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:12.540387 (+ 195us) client_negotiation.cc:769] Sending connection context
0809 05:21:12.540514 (+ 127us) client_negotiation.cc:240] Negotiation successful
0809 05:21:12.540851 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":429,"threads_started":1}
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:12.578624 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41293
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:12.911005 25644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:12.911299 25644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:12.911404 25644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:12.918917 25644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:12.919044 25644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:12.919137 25644 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260809 05:21:12.919191 25644 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:21:12.919263 25644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:12.929400 25644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:41293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:12.930750 25644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:12.932157 25644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:12.945726 25650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:12.945762 25652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:12.945796 25655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:12.946733 25644 server_base.cc:1034] running on GCE node
I20260809 05:21:12.947363 25644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:12.948329 25644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:12.949537 25644 hybrid_clock.cc:648] HybridClock initialized: now 1786252872949458 us; error 73 us; skew 500 ppm
I20260809 05:21:12.949880 25644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:12 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252872, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:12.953050 25644 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:12.954628 25644 webserver.cc:460] Webserver started at http://127.22.1.67:44671/ using document root <none> and password file <none>
I20260809 05:21:12.955351 25644 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:12.955477 25644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:12.955830 25644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:12.958992 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "90018db4012f4390bf8d71c901adeb36"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "e427d5320d8a4d8549c254c98d5acf5f"
server_key_iv: "4575419d92f5ba3bca6cf0229c8df190"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.959792 25644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "90018db4012f4390bf8d71c901adeb36"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "e427d5320d8a4d8549c254c98d5acf5f"
server_key_iv: "4575419d92f5ba3bca6cf0229c8df190"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.965404 25644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 05:21:12.969297 25663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:12.970716 25644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:12.970870 25644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "90018db4012f4390bf8d71c901adeb36"
format_stamp: "Formatted at 2026-08-09 05:21:12 on dist-test-slave-rmhg"
server_key: "e427d5320d8a4d8549c254c98d5acf5f"
server_key_iv: "4575419d92f5ba3bca6cf0229c8df190"
server_key_version: "encryptionkey@0"
I20260809 05:21:12.971069 25644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:12.994266 25644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:12.997530 25427 heartbeater.cc:503] Master 127.22.1.126:41293 was elected leader, sending a full tablet report...
I20260809 05:21:12.998839 25644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:12.999140 25644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:13.002195 25644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:13.002323 25644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:13.002444 25644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:13.002585 25644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 09 05:21:13 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252872, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:13.026810 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.002735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47257 (local address 127.22.1.126:41293)
0809 05:21:13.002919 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.002925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.004669 (+ 1744us) server_negotiation.cc:407] Connection header received
0809 05:21:13.006771 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.006777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.006881 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.006973 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:13.009806 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.011055 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.012826 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.013542 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.013686 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.018748 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:13.018756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:13.018761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:13.018813 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:13.022698 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.023918 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.023925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.023928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.024104 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.024755 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.024759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.024761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.025219 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:13.025366 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.026206 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.026448 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:21:13.027961 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.003450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.67:47257)
0809 05:21:13.004471 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.004542 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.006449 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.007292 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.007309 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.008168 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:13.009528 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.009563 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.011248 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.011260 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.012679 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.012693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.013886 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.013896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.014120 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.014945 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:13.014994 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:13.018418 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:13.022882 (+ 4464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.022899 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.022926 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.023731 (+ 805us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.024233 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.024239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.024243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.024617 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.025371 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:13.025385 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:13.025798 (+ 413us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.026065 (+ 267us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.026667 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":439,"threads_started":1}
I20260809 05:21:13.051609 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.036077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:41781 (local address 127.22.1.126:41293)
0809 05:21:13.036321 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.036329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.036827 (+ 498us) server_negotiation.cc:407] Connection header received
0809 05:21:13.037078 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.037085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.037203 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.037324 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:13.038969 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.039974 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.042205 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.042643 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.042747 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.045221 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:13.045227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:13.045231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:13.045265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:13.048511 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.049444 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.049449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.049453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.049636 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.050059 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.050064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.050068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.050442 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:13.050598 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.051024 (+ 426us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.051241 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:21:13.051720 25776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.036048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.22.1.67:41781)
0809 05:21:13.036710 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.036731 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.036908 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.037382 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.037387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.038122 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:13.038692 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.038716 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.040167 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.040174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.042027 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.042047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.042844 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.042852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.043033 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.043741 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:13.043766 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:13.044963 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:13.048667 (+ 3704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.048674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.048678 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.049296 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.049774 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.049783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.049786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.049942 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.050570 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:13.050575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:13.050770 (+ 195us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.050904 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.051279 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":353,"threads_started":1}
I20260809 05:21:13.057901 25644 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:45263
I20260809 05:21:13.057934 25835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:45263 every 8 connection(s)
I20260809 05:21:13.059175 25644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:13.059751 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 25644
I20260809 05:21:13.059942 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:13.060385 22533 external_mini_cluster.cc:1428] Setting key ce0dff1827a067af63e87ee3a770e575
I20260809 05:21:13.061851 25836 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41293
I20260809 05:21:13.062078 25836 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:13.062732 25836 heartbeater.cc:511] Master 127.22.1.126:41293 requested a full tablet report, sending...
I20260809 05:21:13.064033 25110 ts_manager.cc:194] Registered new tserver with Master: 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263)
I20260809 05:21:13.064878 25110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:47257
I20260809 05:21:13.076336 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 05:21:13 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 05:21:13 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:13.126505 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.112718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.22.1.126:41293)
0809 05:21:13.112897 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.112902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.113104 (+ 202us) server_negotiation.cc:407] Connection header received
0809 05:21:13.113288 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.113292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.113365 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.113492 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:13.114868 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.115655 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.117051 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.117582 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.117690 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.121580 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:13.121585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:13.121588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:13.121626 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:13.124196 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.124844 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.124848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.124850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.124990 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.125363 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.125366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.125368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.125680 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:13.125876 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.126075 (+ 199us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.126253 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:21:13.130964 25110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:13.139047 25110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:13.163071 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.152776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42135 (local address 127.0.0.1:33946)
0809 05:21:13.154520 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.154534 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.154654 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.155360 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.155369 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.155418 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.155881 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.155895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.157863 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.157869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.160139 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.160152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.161968 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.161975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.162201 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.162258 (+ 57us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.162400 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.162632 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1621,"spinlock_wait_cycles":10496,"thread_start_us":334,"threads_started":1}
I20260809 05:21:13.163516 25864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.154703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41361 (local address 127.0.0.1:41482)
0809 05:21:13.155300 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.155316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.155405 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.156662 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.156666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.156693 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.157073 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.157087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.158992 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.158998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.161022 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.161035 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.162673 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.162679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.162853 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.162882 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.162965 (+ 83us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.163098 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":556,"threads_started":1}
I20260809 05:21:13.164127 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.153315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.22.1.66:42135)
0809 05:21:13.154160 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.154167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.154647 (+ 480us) server_negotiation.cc:407] Connection header received
0809 05:21:13.154796 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.154805 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.155101 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.156234 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.156446 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.157708 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.160291 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.161740 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.161867 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.162074 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.163656 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.163858 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":604,"threads_started":1}
I20260809 05:21:13.164491 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.155109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.22.1.65:41361)
0809 05:21:13.155905 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.155914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.155941 (+ 27us) server_negotiation.cc:407] Connection header received
0809 05:21:13.156151 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.156160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.156396 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.157301 (+ 905us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.157529 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.158819 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.161232 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.162449 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.163753 (+ 1304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.163970 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.164038 (+ 68us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.164203 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":1290,"threads_started":1}
I20260809 05:21:13.165469 25859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.152376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45263 (local address 127.0.0.1:53470)
0809 05:21:13.153466 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.153505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.153683 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.157734 (+ 4051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.157739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.157763 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.158228 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.158240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.160552 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.160558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.162513 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.162527 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.164728 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.164734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.164858 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.164881 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.164961 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.165128 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":1900,"threads_started":1}
I20260809 05:21:13.165692 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.152728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.22.1.67:45263)
0809 05:21:13.157124 (+ 4396us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.157130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.157180 (+ 50us) server_negotiation.cc:407] Connection header received
0809 05:21:13.157278 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.157286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.157522 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.157726 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.158666 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.160408 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.162714 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.164279 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.164395 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.164595 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.165034 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.165224 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4248,"thread_start_us":4078,"threads_started":1}
I20260809 05:21:13.168233 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 (DEFAULT_TABLE table=test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.168241 25526 tablet_service.cc:1467] Processing CreateTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 (DEFAULT_TABLE table=test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.168983 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 (DEFAULT_TABLE table=test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.169822 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d8bd7d11fc46fc9178a1fd95493f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.169842 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d8bd7d11fc46fc9178a1fd95493f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.170286 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22d8bd7d11fc46fc9178a1fd95493f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.182399 25866 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Bootstrap starting.
I20260809 05:21:13.184660 25867 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Bootstrap starting.
I20260809 05:21:13.186012 25868 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Bootstrap starting.
I20260809 05:21:13.187341 25866 tablet_bootstrap.cc:654] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.188705 25866 log.cc:824] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:13.189127 25867 tablet_bootstrap.cc:654] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.189419 25868 tablet_bootstrap.cc:654] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.190418 25867 log.cc:824] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:13.190496 25868 log.cc:824] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:13.192205 25866 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: No bootstrap required, opened a new log
I20260809 05:21:13.192523 25866 ts_tablet_manager.cc:1397] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 05:21:13.193244 25868 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: No bootstrap required, opened a new log
I20260809 05:21:13.193490 25868 ts_tablet_manager.cc:1397] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 05:21:13.193521 25867 tablet_bootstrap.cc:492] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: No bootstrap required, opened a new log
I20260809 05:21:13.193769 25867 ts_tablet_manager.cc:1397] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260809 05:21:13.201144 25868 raft_consensus.cc:348] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.201709 25868 raft_consensus.cc:374] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.201812 25868 raft_consensus.cc:729] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fa601ab397415e9397148239fa13f0, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.202459 25868 consensus_queue.cc:260] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.203263 25866 raft_consensus.cc:348] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.203617 25638 heartbeater.cc:503] Master 127.22.1.126:41293 was elected leader, sending a full tablet report...
I20260809 05:21:13.203723 25866 raft_consensus.cc:374] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.203819 25866 raft_consensus.cc:729] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.204380 25868 ts_tablet_manager.cc:1428] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260809 05:21:13.204042 25867 raft_consensus.cc:348] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.204557 25866 consensus_queue.cc:260] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.204763 25867 raft_consensus.cc:374] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.204851 25867 raft_consensus.cc:729] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.205731 25836 heartbeater.cc:503] Master 127.22.1.126:41293 was elected leader, sending a full tablet report...
I20260809 05:21:13.206393 25866 ts_tablet_manager.cc:1428] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260809 05:21:13.206718 25867 consensus_queue.cc:260] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.208479 25867 ts_tablet_manager.cc:1428] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
W20260809 05:21:13.239192 25428 tablet.cc:2369] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:13.261555 25878 raft_consensus.cc:482] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:13.261909 25878 raft_consensus.cc:504] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.263669 25878 leader_election.cc:288] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:13.272053 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.264070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:38511 (local address 127.22.1.67:45263)
0809 05:21:13.264272 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.264279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.264980 (+ 701us) server_negotiation.cc:407] Connection header received
0809 05:21:13.265233 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.265238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.265300 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.265388 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.265959 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.266998 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.269713 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.270952 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.271066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.271178 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.271654 (+ 476us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.271788 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 05:21:13.272071 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.264182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56721 (local address 127.22.1.66:42135)
0809 05:21:13.264357 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.264361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.265379 (+ 1018us) server_negotiation.cc:407] Connection header received
0809 05:21:13.265542 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.265547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.265612 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.265682 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.266283 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.267260 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.269713 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.270903 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.271067 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.271178 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.271654 (+ 476us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.271788 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:21:13.272266 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.264145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45263 (local address 127.22.1.65:38511)
0809 05:21:13.264848 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.264864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.265055 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.265431 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.265436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.265460 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.265800 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.265815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.267141 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.267145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.269525 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.269534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.271213 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.271218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.271395 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.271521 (+ 126us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.271616 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.271804 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":126,"thread_start_us":791,"threads_started":1}
I20260809 05:21:13.272605 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.264168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42135 (local address 127.22.1.65:56721)
0809 05:21:13.265264 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.265278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.265402 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.265739 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.265743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.265773 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.266110 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.266119 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.267378 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.267383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.269513 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.269531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.271180 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.271187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.271451 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.271515 (+ 64us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.271616 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.271804 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":128,"thread_start_us":382,"threads_started":1}
I20260809 05:21:13.273308 25767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d8bd7d11fc46fc9178a1fd95493f50" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36" is_pre_election: true
I20260809 05:21:13.273502 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d8bd7d11fc46fc9178a1fd95493f50" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0" is_pre_election: true
I20260809 05:21:13.273833 25767 raft_consensus.cc:2454] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:13.273978 25557 raft_consensus.cc:2454] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:13.274672 25252 leader_election.cc:302] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 09299ab69a744b29bbd601a0ab09ee35; no voters:
I20260809 05:21:13.275143 25878 raft_consensus.cc:2781] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:13.275269 25878 raft_consensus.cc:482] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:13.275365 25878 raft_consensus.cc:3037] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.277721 25878 raft_consensus.cc:504] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.278326 25878 leader_election.cc:288] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Requested vote from peers 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:13.278852 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d8bd7d11fc46fc9178a1fd95493f50" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0"
I20260809 05:21:13.278852 25767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22d8bd7d11fc46fc9178a1fd95493f50" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36"
I20260809 05:21:13.279062 25767 raft_consensus.cc:3037] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.279062 25557 raft_consensus.cc:3037] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.282284 25557 raft_consensus.cc:2454] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:13.282284 25767 raft_consensus.cc:2454] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:13.282763 25251 leader_election.cc:302] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:13.283094 25878 raft_consensus.cc:2781] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:13.283535 25878 raft_consensus.cc:686] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Becoming Leader. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Running, Role: LEADER
I20260809 05:21:13.284044 25878 consensus_queue.cc:237] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } }
I20260809 05:21:13.288493 25108 catalog_manager.cc:5495] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "09299ab69a744b29bbd601a0ab09ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } health_report { overall_health: HEALTHY } } }
W20260809 05:21:13.290570 25640 tablet.cc:2369] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:13.311643 25837 tablet.cc:2369] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 09 05:21:13 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252871, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:13.365312 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.351771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41293 (local address 127.0.0.1:48860)
0809 05:21:13.352429 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.352445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.352588 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.352893 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.352897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.353672 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:13.353942 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.353953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.354902 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.354905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.355846 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.355858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.356538 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.356543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.356656 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.357340 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:13.357362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:13.359725 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:13.362751 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.362765 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.362773 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.363181 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.363546 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:13.363551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:13.363555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:13.363686 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:13.364186 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:13.364195 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:13.364360 (+ 165us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.364517 (+ 157us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.364799 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":381,"threads_started":1}
I20260809 05:21:13.365316 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.351771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.22.1.126:41293)
0809 05:21:13.352008 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.352012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.352552 (+ 540us) server_negotiation.cc:407] Connection header received
0809 05:21:13.352681 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.352685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.352758 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.352845 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:13.354085 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.354796 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.355963 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.356365 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.356455 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.359908 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:13.359913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:13.359916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:13.359946 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:13.362606 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.363299 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.363305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.363307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.363429 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:13.363788 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:13.363792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:13.363794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:13.364054 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:13.364228 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.364628 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.364951 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":61}
I20260809 05:21:13.370573 25108 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:48860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 05:21:13.371655 25108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:13.378607 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 64d5997b38064b4ebf9c8d40895bab6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5417add7ea29466a95f1ea5f02da872f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:13.378851 25526 tablet_service.cc:1467] Processing CreateTablet for tablet 64d5997b38064b4ebf9c8d40895bab6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5417add7ea29466a95f1ea5f02da872f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:13.378850 25738 tablet_service.cc:1467] Processing CreateTablet for tablet 64d5997b38064b4ebf9c8d40895bab6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5417add7ea29466a95f1ea5f02da872f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:21:13.379566 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d5997b38064b4ebf9c8d40895bab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.379734 25738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d5997b38064b4ebf9c8d40895bab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.379740 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64d5997b38064b4ebf9c8d40895bab6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.384738 25867 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: Bootstrap starting.
I20260809 05:21:13.386322 25867 tablet_bootstrap.cc:654] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.386749 25866 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: Bootstrap starting.
I20260809 05:21:13.387559 25868 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: Bootstrap starting.
I20260809 05:21:13.388391 25866 tablet_bootstrap.cc:654] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.389190 25868 tablet_bootstrap.cc:654] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.390542 25866 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: No bootstrap required, opened a new log
I20260809 05:21:13.390626 25867 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: No bootstrap required, opened a new log
I20260809 05:21:13.390693 25866 ts_tablet_manager.cc:1397] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260809 05:21:13.390776 25867 ts_tablet_manager.cc:1397] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260809 05:21:13.391464 25866 raft_consensus.cc:348] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.391491 25867 raft_consensus.cc:348] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.391722 25866 raft_consensus.cc:374] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.391722 25867 raft_consensus.cc:374] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.391822 25866 raft_consensus.cc:729] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.391817 25867 raft_consensus.cc:729] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.392052 25866 consensus_queue.cc:260] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.392045 25867 consensus_queue.cc:260] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.392508 25878 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.392796 25878 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.392572 25877 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.392946 25877 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.393273 25867 ts_tablet_manager.cc:1428] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:21:13.393644 25866 ts_tablet_manager.cc:1428] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260809 05:21:13.394588 25868 tablet_bootstrap.cc:492] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: No bootstrap required, opened a new log
I20260809 05:21:13.394737 25868 ts_tablet_manager.cc:1397] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260809 05:21:13.395470 25868 raft_consensus.cc:348] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.395717 25868 raft_consensus.cc:374] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.395835 25868 raft_consensus.cc:729] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fa601ab397415e9397148239fa13f0, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.396162 25868 consensus_queue.cc:260] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.396843 25868 ts_tablet_manager.cc:1428] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:21:13.396675 25875 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.397068 25875 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.515738 25877 raft_consensus.cc:482] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:13.515969 25877 raft_consensus.cc:504] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.517688 25877 leader_election.cc:288] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:13.523576 25863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.518107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:40921 (local address 127.22.1.66:42135)
0809 05:21:13.518299 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.518306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.518579 (+ 273us) server_negotiation.cc:407] Connection header received
0809 05:21:13.518813 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.518817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.518897 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.518988 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.519381 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.520152 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.521885 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.522661 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.522773 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.522862 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.523162 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.523283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:21:13.523566 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.518126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42135 (local address 127.22.1.67:40921)
0809 05:21:13.518477 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.518498 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.518668 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.519033 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.519039 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.519065 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.519254 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.519265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.520283 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.520289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.521716 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.521731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.522816 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.522822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.523000 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.523026 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.523120 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.523271 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260809 05:21:13.523804 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.518103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:43617 (local address 127.22.1.65:41361)
0809 05:21:13.518336 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.518342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.518913 (+ 571us) server_negotiation.cc:407] Connection header received
0809 05:21:13.519074 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.519078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.519149 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.519223 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:13.519774 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.520457 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.522144 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.522898 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.523021 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.523124 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.523441 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.523575 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260809 05:21:13.523924 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.518177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41361 (local address 127.22.1.67:43617)
0809 05:21:13.518806 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.518823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.518933 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.519285 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.519290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.519314 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:13.519619 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.519628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.520602 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.520607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.521975 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.521988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.523135 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.523142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.523277 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.523300 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.523382 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.523510 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":316,"threads_started":1}
I20260809 05:21:13.524369 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d5997b38064b4ebf9c8d40895bab6e" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0" is_pre_election: true
I20260809 05:21:13.524652 25557 raft_consensus.cc:2454] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 0.
I20260809 05:21:13.524920 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d5997b38064b4ebf9c8d40895bab6e" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09299ab69a744b29bbd601a0ab09ee35" is_pre_election: true
I20260809 05:21:13.525249 25666 leader_election.cc:302] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:13.525403 25358 raft_consensus.cc:2454] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 0.
I20260809 05:21:13.525709 25877 raft_consensus.cc:2781] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:13.525827 25877 raft_consensus.cc:482] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:13.525926 25877 raft_consensus.cc:3037] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.528095 25877 raft_consensus.cc:504] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.528678 25877 leader_election.cc:288] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 election: Requested vote from peers 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:13.529234 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d5997b38064b4ebf9c8d40895bab6e" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09299ab69a744b29bbd601a0ab09ee35"
I20260809 05:21:13.529220 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64d5997b38064b4ebf9c8d40895bab6e" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0"
I20260809 05:21:13.529422 25358 raft_consensus.cc:3037] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.529446 25557 raft_consensus.cc:3037] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:13.532440 25557 raft_consensus.cc:2454] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 1.
I20260809 05:21:13.532450 25358 raft_consensus.cc:2454] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 1.
I20260809 05:21:13.532944 25665 leader_election.cc:302] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:13.533293 25877 raft_consensus.cc:2781] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:13.533768 25877 raft_consensus.cc:686] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [term 1 LEADER]: Becoming Leader. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Running, Role: LEADER
I20260809 05:21:13.534265 25877 consensus_queue.cc:237] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.537716 25899 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: TxnStatusTablet state changed. Reason: New leader 90018db4012f4390bf8d71c901adeb36. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.538096 25899 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:13.538801 25901 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 05:21:13.539074 25901 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 05:21:13.539642 25110 catalog_manager.cc:5495] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90018db4012f4390bf8d71c901adeb36 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } health_report { overall_health: HEALTHY } } }
I20260809 05:21:13.591548 25892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.584872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45263 (local address 127.0.0.1:53482)
0809 05:21:13.585116 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.585132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.585298 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.585856 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.585860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.585871 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:13.586046 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.586054 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.587529 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.587534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.588572 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.588584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.589604 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.589608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.589838 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.589916 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.591067 (+ 1151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.591091 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.591155 (+ 64us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.591289 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 05:21:13.591862 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.584829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53482 (local address 127.22.1.67:45263)
0809 05:21:13.585026 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.585032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.585236 (+ 204us) server_negotiation.cc:407] Connection header received
0809 05:21:13.585515 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.585521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.585667 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.585842 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:13.586185 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.587382 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.588731 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.589415 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.589603 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.590114 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.590936 (+ 822us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.591084 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.591211 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.591313 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:21:13.599781 25557 raft_consensus.cc:1261] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Refusing update from remote peer 90018db4012f4390bf8d71c901adeb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:13.599756 25358 raft_consensus.cc:1261] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Refusing update from remote peer 90018db4012f4390bf8d71c901adeb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:13.600716 25877 consensus_queue.cc:1035] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:13.601032 25899 consensus_queue.cc:1035] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:13.615793 25875 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: TxnStatusTablet state changed. Reason: New leader 90018db4012f4390bf8d71c901adeb36. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.616114 25875 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.617404 25878 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: TxnStatusTablet state changed. Reason: New leader 90018db4012f4390bf8d71c901adeb36. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.617790 25878 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.622399 25910 mvcc.cc:205] Tried to move back new op lower bound from 7316491770254336000 to 7316491770000322560. Current Snapshot: MvccSnapshot[applied={T|T < 7316491770254336000}]
I20260809 05:21:13.623387 25899 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.623690 25899 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:13.634140 25899 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.634476 25899 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 90018db4012f4390bf8d71c901adeb36: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:21:13.638005 25875 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.638279 25875 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 07fa601ab397415e9397148239fa13f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.642557 25878 tablet_replica.cc:439] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } } }
I20260809 05:21:13.642870 25878 tablet_replica.cc:442] T 64d5997b38064b4ebf9c8d40895bab6e P 09299ab69a744b29bbd601a0ab09ee35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:21:13.651540 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.642407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.22.1.65:41361)
0809 05:21:13.643628 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.643634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.645494 (+ 1860us) server_negotiation.cc:407] Connection header received
0809 05:21:13.645710 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.645713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.645809 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.645930 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:13.646443 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.647178 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.649191 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.649609 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.649700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.650253 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.650867 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.650986 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.651112 (+ 126us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.651183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088}
I20260809 05:21:13.668753 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.662900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:47873 (local address 127.22.1.67:45263)
0809 05:21:13.663074 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.663079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.663634 (+ 555us) server_negotiation.cc:407] Connection header received
0809 05:21:13.663890 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.663895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.664005 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.664068 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:13.664722 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.665531 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.666986 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.667441 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.667567 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.668043 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.668151 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.668246 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.668368 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.668456 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:21:13.668874 25925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.662907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45263 (local address 127.22.1.65:47873)
0809 05:21:13.663520 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.663536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.663672 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.664238 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.664241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.664251 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:13.664556 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.664564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.665661 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.665666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.666843 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.666856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.667638 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.667643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.667761 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.667822 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.668247 (+ 425us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.668260 (+ 13us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.668334 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.668487 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":319,"threads_started":1}
I20260809 05:21:13.672251 25878 consensus_queue.cc:1035] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:13.679118 25557 raft_consensus.cc:1261] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Refusing update from remote peer 09299ab69a744b29bbd601a0ab09ee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:13.679672 25878 consensus_queue.cc:1035] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:13.717643 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.711271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:44177 (local address 127.22.1.65:41361)
0809 05:21:13.711445 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:13.711452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:13.711619 (+ 167us) server_negotiation.cc:407] Connection header received
0809 05:21:13.711906 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:13.711912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:13.711989 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:13.712071 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:13.712755 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.713901 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.715564 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.716108 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.716230 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.716859 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.716997 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.717090 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:13.717245 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 05:21:13.717336 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:21:13.717684 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:13.710754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41361 (local address 127.22.1.67:44177)
0809 05:21:13.711501 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:13.711532 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:13.711736 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:13.712109 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:13.712114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:13.712126 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:13.712519 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.712530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.714033 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.714037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.715383 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:13.715399 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:13.716364 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:13.716371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:13.716564 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:13.716649 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:13.717105 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:13.717114 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 05:21:13.717217 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 05:21:13.717332 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":726,"threads_started":1}
I20260809 05:21:13.759415 25108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:13.759769 25108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:13.764075 25108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Sending DeleteTablet for 3 replicas of tablet 22d8bd7d11fc46fc9178a1fd95493f50
I20260809 05:21:13.765452 25526 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:13 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:33946
I20260809 05:21:13.765473 25738 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:13 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53470
I20260809 05:21:13.767342 25322 tablet_service.cc:1514] Processing DeleteTablet for tablet 22d8bd7d11fc46fc9178a1fd95493f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:13 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41482
I20260809 05:21:13.773138 25941 tablet_replica.cc:330] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: stopping tablet replica
I20260809 05:21:13.773738 25941 raft_consensus.cc:2229] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:13.774150 25108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:13.775023 25108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:13.783946 25525 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 (DEFAULT_TABLE table=test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.774510 25941 raft_consensus.cc:2258] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:13.784474 25737 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 (DEFAULT_TABLE table=test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.784653 25525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce6af5e8495462fb739aa9df6b38689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.784610 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 (DEFAULT_TABLE table=test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:13.785081 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce6af5e8495462fb739aa9df6b38689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.785107 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce6af5e8495462fb739aa9df6b38689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:13.786893 25941 ts_tablet_manager.cc:1905] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:13.791307 25868 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Bootstrap starting.
I20260809 05:21:13.793056 25868 tablet_bootstrap.cc:654] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.793619 25866 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Bootstrap starting.
I20260809 05:21:13.795277 25866 tablet_bootstrap.cc:654] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.795282 25941 ts_tablet_manager.cc:1918] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:13.797397 25941 log.cc:1196] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/22d8bd7d11fc46fc9178a1fd95493f50
I20260809 05:21:13.798105 25941 ts_tablet_manager.cc:1939] T 22d8bd7d11fc46fc9178a1fd95493f50 P 09299ab69a744b29bbd601a0ab09ee35: Deleting consensus metadata
I20260809 05:21:13.798676 25867 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Bootstrap starting.
I20260809 05:21:13.800424 25867 tablet_bootstrap.cc:654] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:13.801992 25087 catalog_manager.cc:4841] TS 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361): tablet 22d8bd7d11fc46fc9178a1fd95493f50 (table test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]) successfully deleted
I20260809 05:21:13.803016 25938 tablet_replica.cc:330] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: stopping tablet replica
I20260809 05:21:13.803545 25938 raft_consensus.cc:2229] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:13.803911 25938 raft_consensus.cc:2258] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:13.804193 25866 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: No bootstrap required, opened a new log
I20260809 05:21:13.804366 25866 ts_tablet_manager.cc:1397] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20260809 05:21:13.805312 25868 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: No bootstrap required, opened a new log
I20260809 05:21:13.805248 25866 raft_consensus.cc:348] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.805518 25866 raft_consensus.cc:374] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.805613 25866 raft_consensus.cc:729] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.805642 25868 ts_tablet_manager.cc:1397] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20260809 05:21:13.806376 25868 raft_consensus.cc:348] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.806609 25868 raft_consensus.cc:374] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.806675 25868 raft_consensus.cc:729] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fa601ab397415e9397148239fa13f0, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.806933 25868 consensus_queue.cc:260] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.807127 25866 consensus_queue.cc:260] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.808382 25866 ts_tablet_manager.cc:1428] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260809 05:21:13.809294 25937 tablet_replica.cc:330] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: stopping tablet replica
I20260809 05:21:13.811546 25868 ts_tablet_manager.cc:1428] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260809 05:21:13.814208 25937 raft_consensus.cc:2229] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:13.814636 25937 raft_consensus.cc:2258] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:13.814990 25938 ts_tablet_manager.cc:1905] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:13.817652 25937 ts_tablet_manager.cc:1905] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:13.820040 25867 tablet_bootstrap.cc:492] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: No bootstrap required, opened a new log
I20260809 05:21:13.820231 25867 ts_tablet_manager.cc:1397] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.004s
I20260809 05:21:13.821034 25867 raft_consensus.cc:348] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.821352 25867 raft_consensus.cc:374] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:13.821444 25867 raft_consensus.cc:729] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Initialized, Role: FOLLOWER
I20260809 05:21:13.821727 25867 consensus_queue.cc:260] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:13.822522 25938 ts_tablet_manager.cc:1918] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:13.822731 25938 log.cc:1196] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/22d8bd7d11fc46fc9178a1fd95493f50
I20260809 05:21:13.823324 25938 ts_tablet_manager.cc:1939] T 22d8bd7d11fc46fc9178a1fd95493f50 P 90018db4012f4390bf8d71c901adeb36: Deleting consensus metadata
I20260809 05:21:13.824961 25867 ts_tablet_manager.cc:1428] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260809 05:21:13.825238 25087 catalog_manager.cc:4841] TS 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263): tablet 22d8bd7d11fc46fc9178a1fd95493f50 (table test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]) successfully deleted
I20260809 05:21:13.828229 25937 ts_tablet_manager.cc:1918] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:13.828428 25937 log.cc:1196] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/22d8bd7d11fc46fc9178a1fd95493f50
I20260809 05:21:13.828965 25937 ts_tablet_manager.cc:1939] T 22d8bd7d11fc46fc9178a1fd95493f50 P 07fa601ab397415e9397148239fa13f0: Deleting consensus metadata
I20260809 05:21:13.830927 25088 catalog_manager.cc:4841] TS 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135): tablet 22d8bd7d11fc46fc9178a1fd95493f50 (table test-table [id=57bf0a50e12f4505b4ecba3323df4ec3]) successfully deleted
I20260809 05:21:14.030938 25915 raft_consensus.cc:482] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:14.031176 25915 raft_consensus.cc:504] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:14.031926 25915 leader_election.cc:288] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:14.032752 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce6af5e8495462fb739aa9df6b38689" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09299ab69a744b29bbd601a0ab09ee35" is_pre_election: true
I20260809 05:21:14.032757 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce6af5e8495462fb739aa9df6b38689" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0" is_pre_election: true
I20260809 05:21:14.033103 25557 raft_consensus.cc:2454] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 0.
I20260809 05:21:14.033121 25358 raft_consensus.cc:2454] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 0.
I20260809 05:21:14.033727 25666 leader_election.cc:302] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:14.034293 25915 raft_consensus.cc:2781] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:14.034582 25915 raft_consensus.cc:482] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:14.034813 25915 raft_consensus.cc:3037] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:14.038048 25915 raft_consensus.cc:504] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:14.038821 25915 leader_election.cc:288] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 election: Requested vote from peers 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361), 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135)
I20260809 05:21:14.039601 25358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce6af5e8495462fb739aa9df6b38689" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09299ab69a744b29bbd601a0ab09ee35"
I20260809 05:21:14.039852 25358 raft_consensus.cc:3037] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:14.040072 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce6af5e8495462fb739aa9df6b38689" candidate_uuid: "90018db4012f4390bf8d71c901adeb36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0"
I20260809 05:21:14.040370 25557 raft_consensus.cc:3037] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:14.043866 25358 raft_consensus.cc:2454] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 1.
I20260809 05:21:14.044339 25665 leader_election.cc:302] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:14.044713 25915 raft_consensus.cc:2781] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:14.045181 25557 raft_consensus.cc:2454] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90018db4012f4390bf8d71c901adeb36 in term 1.
I20260809 05:21:14.045291 25915 raft_consensus.cc:686] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 1 LEADER]: Becoming Leader. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Running, Role: LEADER
I20260809 05:21:14.045689 25915 consensus_queue.cc:237] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:14.049412 25110 catalog_manager.cc:5495] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90018db4012f4390bf8d71c901adeb36 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "90018db4012f4390bf8d71c901adeb36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } health_report { overall_health: HEALTHY } } }
I20260809 05:21:14.122757 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:14.116812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.22.1.67:45263)
0809 05:21:14.116995 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:14.117002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:14.117093 (+ 91us) server_negotiation.cc:407] Connection header received
0809 05:21:14.117333 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:14.117337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:14.117404 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:14.117482 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:14.117918 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.118810 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.120433 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.121283 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.121399 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:14.121996 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:14.122139 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:14.122219 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:14.122392 (+ 173us) server_negotiation.cc:299] Negotiation successful
0809 05:21:14.122475 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:21:14.139951 25860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:14.133415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:39771 (local address 127.22.1.67:45263)
0809 05:21:14.133622 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:14.133628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:14.133648 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:21:14.133746 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:14.133751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:14.133831 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:14.133947 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:14.134498 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.135877 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.137519 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.138302 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.138455 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:14.139094 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:14.139229 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:14.139316 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:14.139509 (+ 193us) server_negotiation.cc:299] Negotiation successful
0809 05:21:14.139630 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":90}
I20260809 05:21:14.140301 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:14.133355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45263 (local address 127.22.1.67:39771)
0809 05:21:14.133538 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:14.133559 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:14.133651 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:14.133955 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:14.133958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:14.133967 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:21:14.134313 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:14.134322 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.136102 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.136107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:14.137357 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:14.137370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:14.138550 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:14.138559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:14.138852 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:14.138908 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:21:14.139315 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:21:14.139358 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 05:21:14.139469 (+ 111us) client_negotiation.cc:240] Negotiation successful
0809 05:21:14.139729 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":33}
I20260809 05:21:14.149257 25557 raft_consensus.cc:1261] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Refusing update from remote peer 90018db4012f4390bf8d71c901adeb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:14.149747 25358 raft_consensus.cc:1261] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Refusing update from remote peer 90018db4012f4390bf8d71c901adeb36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:14.150002 25899 consensus_queue.cc:1035] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:14.150352 25915 consensus_queue.cc:1035] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:14.157510 25929 mvcc.cc:205] Tried to move back new op lower bound from 7316491772509626368 to 7316491772094054400. Current Snapshot: MvccSnapshot[applied={T|T < 7316491772509626368}]
W20260809 05:21:14.177954 25929 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 05:21:15.784044 25108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:15.784344 25108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:15.789691 25108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Sending DeleteTablet for 3 replicas of tablet 2ce6af5e8495462fb739aa9df6b38689
I20260809 05:21:15.790663 25322 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:15 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41482
I20260809 05:21:15.790772 25526 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:15 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:33946
I20260809 05:21:15.790796 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce6af5e8495462fb739aa9df6b38689 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:15 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53470
I20260809 05:21:15.800267 25984 tablet_replica.cc:330] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: stopping tablet replica
I20260809 05:21:15.800760 25984 raft_consensus.cc:2229] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:15.801380 25984 raft_consensus.cc:2258] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:15.802728 25984 ts_tablet_manager.cc:1905] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:15.806115 25983 tablet_replica.cc:330] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: stopping tablet replica
I20260809 05:21:15.806193 25982 tablet_replica.cc:330] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: stopping tablet replica
I20260809 05:21:15.806331 25983 raft_consensus.cc:2229] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:15.806468 25982 raft_consensus.cc:2229] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:15.806833 25982 raft_consensus.cc:2258] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:15.807078 25983 raft_consensus.cc:2258] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:15.808243 25983 ts_tablet_manager.cc:1905] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:15.808269 25982 ts_tablet_manager.cc:1905] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:15.811455 25984 ts_tablet_manager.cc:1918] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:15.811691 25984 log.cc:1196] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/2ce6af5e8495462fb739aa9df6b38689
I20260809 05:21:15.812493 25984 ts_tablet_manager.cc:1939] T 2ce6af5e8495462fb739aa9df6b38689 P 07fa601ab397415e9397148239fa13f0: Deleting consensus metadata
I20260809 05:21:15.813735 25088 catalog_manager.cc:4841] TS 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135): tablet 2ce6af5e8495462fb739aa9df6b38689 (table test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]) successfully deleted
I20260809 05:21:15.816769 25982 ts_tablet_manager.cc:1918] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:15.817031 25982 log.cc:1196] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/2ce6af5e8495462fb739aa9df6b38689
I20260809 05:21:15.817871 25982 ts_tablet_manager.cc:1939] T 2ce6af5e8495462fb739aa9df6b38689 P 09299ab69a744b29bbd601a0ab09ee35: Deleting consensus metadata
I20260809 05:21:15.818820 25983 ts_tablet_manager.cc:1918] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:15.819152 25983 log.cc:1196] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/2ce6af5e8495462fb739aa9df6b38689
I20260809 05:21:15.819111 25087 catalog_manager.cc:4841] TS 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361): tablet 2ce6af5e8495462fb739aa9df6b38689 (table test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]) successfully deleted
I20260809 05:21:15.819609 25983 ts_tablet_manager.cc:1939] T 2ce6af5e8495462fb739aa9df6b38689 P 90018db4012f4390bf8d71c901adeb36: Deleting consensus metadata
I20260809 05:21:15.820580 25087 catalog_manager.cc:4841] TS 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263): tablet 2ce6af5e8495462fb739aa9df6b38689 (table test-table [id=ed5f6e451a9245a5bf9fa1ff4ae7f44a]) successfully deleted
Aug 09 05:21:15 dist-test-slave-rmhg krb5kdc[25047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 05:21:15 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:15.859853 25994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:15.842931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.22.1.126:41293)
0809 05:21:15.843539 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:15.843545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:15.843572 (+ 27us) server_negotiation.cc:407] Connection header received
0809 05:21:15.843696 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:15.843701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:15.843767 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:15.843876 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:15.845198 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:15.845982 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:15.847180 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:15.847571 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:15.847666 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:15.853007 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:15.853014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:15.853018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:15.853057 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:15.856989 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:15.857789 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:15.857793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:15.857796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:15.857991 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:15.858362 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:15.858365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:15.858367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:15.858718 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:15.858951 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:15.859181 (+ 230us) server_negotiation.cc:299] Negotiation successful
0809 05:21:15.859412 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
I20260809 05:21:15.863724 25108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:15.864490 25108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:15.871382 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 4f15254effa04e25aac725705a8c3a69 (DEFAULT_TABLE table=test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:15.871534 25526 tablet_service.cc:1467] Processing CreateTablet for tablet 4f15254effa04e25aac725705a8c3a69 (DEFAULT_TABLE table=test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:15.871752 25737 tablet_service.cc:1467] Processing CreateTablet for tablet 4f15254effa04e25aac725705a8c3a69 (DEFAULT_TABLE table=test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:15.871923 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f15254effa04e25aac725705a8c3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:15.872006 25526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f15254effa04e25aac725705a8c3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:15.872324 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f15254effa04e25aac725705a8c3a69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:15.879535 25995 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Bootstrap starting.
I20260809 05:21:15.880908 25996 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Bootstrap starting.
I20260809 05:21:15.881803 25997 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Bootstrap starting.
I20260809 05:21:15.883706 25995 tablet_bootstrap.cc:654] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:15.883872 25996 tablet_bootstrap.cc:654] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:15.884624 25997 tablet_bootstrap.cc:654] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:15.888178 25995 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: No bootstrap required, opened a new log
I20260809 05:21:15.888420 25995 ts_tablet_manager.cc:1397] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260809 05:21:15.889051 25996 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: No bootstrap required, opened a new log
I20260809 05:21:15.889318 25996 ts_tablet_manager.cc:1397] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260809 05:21:15.889916 25997 tablet_bootstrap.cc:492] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: No bootstrap required, opened a new log
I20260809 05:21:15.889890 25995 raft_consensus.cc:348] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.890218 25997 ts_tablet_manager.cc:1397] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260809 05:21:15.890276 25995 raft_consensus.cc:374] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:15.890502 25995 raft_consensus.cc:729] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Initialized, Role: FOLLOWER
I20260809 05:21:15.890587 25996 raft_consensus.cc:348] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.890849 25996 raft_consensus.cc:374] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:15.890988 25996 raft_consensus.cc:729] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fa601ab397415e9397148239fa13f0, State: Initialized, Role: FOLLOWER
I20260809 05:21:15.891292 25996 consensus_queue.cc:260] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.891098 25995 consensus_queue.cc:260] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.891539 25997 raft_consensus.cc:348] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.891852 25997 raft_consensus.cc:374] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:15.891980 25997 raft_consensus.cc:729] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Initialized, Role: FOLLOWER
I20260809 05:21:15.892256 25997 consensus_queue.cc:260] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:15.892908 25997 ts_tablet_manager.cc:1428] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260809 05:21:15.894115 25995 ts_tablet_manager.cc:1428] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260809 05:21:15.894356 25996 ts_tablet_manager.cc:1428] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260809 05:21:16.154838 26005 raft_consensus.cc:482] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:16.155247 26005 raft_consensus.cc:504] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.156280 26005 leader_election.cc:288] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135), 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263)
I20260809 05:21:16.156991 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f15254effa04e25aac725705a8c3a69" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0" is_pre_election: true
I20260809 05:21:16.156991 25765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f15254effa04e25aac725705a8c3a69" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36" is_pre_election: true
I20260809 05:21:16.157277 25557 raft_consensus.cc:2454] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:16.157277 25765 raft_consensus.cc:2454] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:16.157828 25251 leader_election.cc:302] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:16.158178 26005 raft_consensus.cc:2781] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:16.158419 26005 raft_consensus.cc:482] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:16.158605 26005 raft_consensus.cc:3037] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.161990 26005 raft_consensus.cc:504] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.162683 26005 leader_election.cc:288] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Requested vote from peers 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135), 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263)
I20260809 05:21:16.163300 25765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f15254effa04e25aac725705a8c3a69" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36"
I20260809 05:21:16.163388 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f15254effa04e25aac725705a8c3a69" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0"
I20260809 05:21:16.163569 25765 raft_consensus.cc:3037] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.163672 25557 raft_consensus.cc:3037] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.166486 25557 raft_consensus.cc:2454] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:16.166611 25765 raft_consensus.cc:2454] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:16.166934 25252 leader_election.cc:302] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 09299ab69a744b29bbd601a0ab09ee35; no voters:
I20260809 05:21:16.167253 26005 raft_consensus.cc:2781] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:16.167989 26005 raft_consensus.cc:686] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Becoming Leader. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Running, Role: LEADER
I20260809 05:21:16.168262 26005 consensus_queue.cc:237] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.171263 25110 catalog_manager.cc:5495] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "09299ab69a744b29bbd601a0ab09ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:16.263186 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:16.255735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.22.1.65:41361)
0809 05:21:16.256722 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:16.256726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:16.256745 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:21:16.256795 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:16.256799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:16.256857 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:16.256942 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:16.257541 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:16.258786 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:16.260704 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:16.261431 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:16.261541 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:16.262160 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:16.262300 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:16.262379 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:16.262541 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 05:21:16.262617 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":838,"thread_start_us":537,"threads_started":1}
I20260809 05:21:16.274045 25765 raft_consensus.cc:1261] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Refusing update from remote peer 09299ab69a744b29bbd601a0ab09ee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:16.274171 25557 raft_consensus.cc:1261] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Refusing update from remote peer 09299ab69a744b29bbd601a0ab09ee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:16.274806 26006 consensus_queue.cc:1035] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:16.275359 26005 consensus_queue.cc:1035] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:16.281488 25960 mvcc.cc:205] Tried to move back new op lower bound from 7316491781213257728 to 7316491780787572736. Current Snapshot: MvccSnapshot[applied={T|T < 7316491781213257728}]
I20260809 05:21:16.345958 25109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:16.346333 25109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:16.349951 25109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Sending DeleteTablet for 3 replicas of tablet 4f15254effa04e25aac725705a8c3a69
I20260809 05:21:16.350878 25322 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f15254effa04e25aac725705a8c3a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:16 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41482
I20260809 05:21:16.351011 25526 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f15254effa04e25aac725705a8c3a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:16 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:33946
I20260809 05:21:16.351929 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f15254effa04e25aac725705a8c3a69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:16 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53470
I20260809 05:21:16.352213 25109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:16.353219 25109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:16.356964 26019 tablet_replica.cc:330] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: stopping tablet replica
I20260809 05:21:16.357235 26019 raft_consensus.cc:2229] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:16.357702 26019 raft_consensus.cc:2258] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:16.358449 26024 tablet_replica.cc:330] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: stopping tablet replica
I20260809 05:21:16.358843 26024 raft_consensus.cc:2229] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:16.359170 26024 raft_consensus.cc:2258] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:16.360172 26019 ts_tablet_manager.cc:1905] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:16.362500 26024 ts_tablet_manager.cc:1905] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:16.363554 25525 tablet_service.cc:1467] Processing CreateTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 (DEFAULT_TABLE table=test-table [id=7ea9ccde4c57493e8869b5add4b8b365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:16.363898 25321 tablet_service.cc:1467] Processing CreateTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 (DEFAULT_TABLE table=test-table [id=7ea9ccde4c57493e8869b5add4b8b365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:16.364035 25525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08c17d0a3e54390a3b2847f59ba8c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:16.364362 25321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08c17d0a3e54390a3b2847f59ba8c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:16.368978 26019 ts_tablet_manager.cc:1918] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:16.369220 26019 log.cc:1196] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/4f15254effa04e25aac725705a8c3a69
I20260809 05:21:16.369711 25737 tablet_service.cc:1467] Processing CreateTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 (DEFAULT_TABLE table=test-table [id=7ea9ccde4c57493e8869b5add4b8b365]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:16.369882 26024 ts_tablet_manager.cc:1918] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:16.370035 26019 ts_tablet_manager.cc:1939] T 4f15254effa04e25aac725705a8c3a69 P 09299ab69a744b29bbd601a0ab09ee35: Deleting consensus metadata
I20260809 05:21:16.370180 26024 log.cc:1196] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/4f15254effa04e25aac725705a8c3a69
I20260809 05:21:16.370389 25737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08c17d0a3e54390a3b2847f59ba8c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:16.371250 25087 catalog_manager.cc:4841] TS 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361): tablet 4f15254effa04e25aac725705a8c3a69 (table test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]) successfully deleted
I20260809 05:21:16.373656 25995 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Bootstrap starting.
I20260809 05:21:16.374382 26021 tablet_replica.cc:330] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: stopping tablet replica
I20260809 05:21:16.375759 26021 raft_consensus.cc:2229] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:16.375876 25996 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Bootstrap starting.
I20260809 05:21:16.376065 26021 raft_consensus.cc:2258] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:16.376292 25995 tablet_bootstrap.cc:654] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:16.377404 26024 ts_tablet_manager.cc:1939] T 4f15254effa04e25aac725705a8c3a69 P 90018db4012f4390bf8d71c901adeb36: Deleting consensus metadata
I20260809 05:21:16.378934 25996 tablet_bootstrap.cc:654] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:16.381614 26021 ts_tablet_manager.cc:1905] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:16.383152 25087 catalog_manager.cc:4841] TS 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263): tablet 4f15254effa04e25aac725705a8c3a69 (table test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]) successfully deleted
I20260809 05:21:16.384192 25995 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: No bootstrap required, opened a new log
I20260809 05:21:16.384405 25995 ts_tablet_manager.cc:1397] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260809 05:21:16.385442 25996 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: No bootstrap required, opened a new log
I20260809 05:21:16.385718 25996 ts_tablet_manager.cc:1397] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260809 05:21:16.385630 25995 raft_consensus.cc:348] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.386096 25995 raft_consensus.cc:374] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:16.386276 25995 raft_consensus.cc:729] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Initialized, Role: FOLLOWER
I20260809 05:21:16.386682 25995 consensus_queue.cc:260] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.386806 25996 raft_consensus.cc:348] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.387216 25996 raft_consensus.cc:374] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:16.387387 25996 raft_consensus.cc:729] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fa601ab397415e9397148239fa13f0, State: Initialized, Role: FOLLOWER
I20260809 05:21:16.387414 25995 ts_tablet_manager.cc:1428] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260809 05:21:16.387791 25996 consensus_queue.cc:260] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.390208 26021 ts_tablet_manager.cc:1918] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:21:16.390362 26021 log.cc:1196] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/4f15254effa04e25aac725705a8c3a69
I20260809 05:21:16.390861 26021 ts_tablet_manager.cc:1939] T 4f15254effa04e25aac725705a8c3a69 P 07fa601ab397415e9397148239fa13f0: Deleting consensus metadata
I20260809 05:21:16.394322 25996 ts_tablet_manager.cc:1428] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20260809 05:21:16.394481 26027 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Bootstrap starting.
I20260809 05:21:16.395080 25088 catalog_manager.cc:4841] TS 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135): tablet 4f15254effa04e25aac725705a8c3a69 (table test-table [id=029892a3dbb440aa9ecf6a0300ed6ec7]) successfully deleted
I20260809 05:21:16.397011 26027 tablet_bootstrap.cc:654] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:16.401106 26027 tablet_bootstrap.cc:492] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: No bootstrap required, opened a new log
I20260809 05:21:16.401392 26027 ts_tablet_manager.cc:1397] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260809 05:21:16.402650 26027 raft_consensus.cc:348] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.402961 26027 raft_consensus.cc:374] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:16.403187 26027 raft_consensus.cc:729] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90018db4012f4390bf8d71c901adeb36, State: Initialized, Role: FOLLOWER
I20260809 05:21:16.403494 26027 consensus_queue.cc:260] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.404102 26027 ts_tablet_manager.cc:1428] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260809 05:21:16.494292 26017 raft_consensus.cc:482] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:16.494578 26017 raft_consensus.cc:504] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.495380 26017 leader_election.cc:288] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135), 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263)
I20260809 05:21:16.496111 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c17d0a3e54390a3b2847f59ba8c97" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0" is_pre_election: true
I20260809 05:21:16.496150 25765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c17d0a3e54390a3b2847f59ba8c97" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36" is_pre_election: true
I20260809 05:21:16.496456 25765 raft_consensus.cc:2454] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:16.496476 25557 raft_consensus.cc:2454] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 0.
I20260809 05:21:16.497009 25251 leader_election.cc:302] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:16.497437 26017 raft_consensus.cc:2781] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:16.497651 26017 raft_consensus.cc:482] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:16.497795 26017 raft_consensus.cc:3037] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.500458 26017 raft_consensus.cc:504] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.501363 26017 leader_election.cc:288] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Requested vote from peers 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135), 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263)
I20260809 05:21:16.501811 25765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c17d0a3e54390a3b2847f59ba8c97" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90018db4012f4390bf8d71c901adeb36"
I20260809 05:21:16.501904 25557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08c17d0a3e54390a3b2847f59ba8c97" candidate_uuid: "09299ab69a744b29bbd601a0ab09ee35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fa601ab397415e9397148239fa13f0"
I20260809 05:21:16.502048 25765 raft_consensus.cc:3037] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.502205 25557 raft_consensus.cc:3037] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:16.504773 25765 raft_consensus.cc:2454] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:16.504801 25557 raft_consensus.cc:2454] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09299ab69a744b29bbd601a0ab09ee35 in term 1.
I20260809 05:21:16.505270 25251 leader_election.cc:302] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fa601ab397415e9397148239fa13f0, 09299ab69a744b29bbd601a0ab09ee35, 90018db4012f4390bf8d71c901adeb36; no voters:
I20260809 05:21:16.505622 26017 raft_consensus.cc:2781] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:16.505813 26017 raft_consensus.cc:686] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Becoming Leader. State: Replica: 09299ab69a744b29bbd601a0ab09ee35, State: Running, Role: LEADER
I20260809 05:21:16.506119 26017 consensus_queue.cc:237] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } }
I20260809 05:21:16.509086 25110 catalog_manager.cc:5495] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "09299ab69a744b29bbd601a0ab09ee35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09299ab69a744b29bbd601a0ab09ee35" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:16.555642 25765 raft_consensus.cc:1261] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Refusing update from remote peer 09299ab69a744b29bbd601a0ab09ee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:16.555636 25557 raft_consensus.cc:1261] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Refusing update from remote peer 09299ab69a744b29bbd601a0ab09ee35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:16.556195 26017 consensus_queue.cc:1035] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90018db4012f4390bf8d71c901adeb36" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:16.556592 26005 consensus_queue.cc:1035] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fa601ab397415e9397148239fa13f0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 05:21:16.579907 25960 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260809 05:21:18.195019 25108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:18.195389 25108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:18.201084 25108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80f4375944d40d2868ff543d9b7f66d: Sending DeleteTablet for 3 replicas of tablet f08c17d0a3e54390a3b2847f59ba8c97
I20260809 05:21:18.202260 25321 tablet_service.cc:1514] Processing DeleteTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:18 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41482
I20260809 05:21:18.202334 25525 tablet_service.cc:1514] Processing DeleteTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:18 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:33946
I20260809 05:21:18.202302 25737 tablet_service.cc:1514] Processing DeleteTablet for tablet f08c17d0a3e54390a3b2847f59ba8c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:18 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53470
I20260809 05:21:18.204789 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25225
I20260809 05:21:18.215062 26073 tablet_replica.cc:330] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: stopping tablet replica
I20260809 05:21:18.215124 26071 tablet_replica.cc:330] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: stopping tablet replica
I20260809 05:21:18.215386 26071 raft_consensus.cc:2229] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:18.215556 26073 raft_consensus.cc:2229] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:18.216032 26071 raft_consensus.cc:2258] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:18.216210 26073 raft_consensus.cc:2258] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:18.217351 26071 ts_tablet_manager.cc:1905] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:18.217412 26073 ts_tablet_manager.cc:1905] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:18.218854 26070 tablet_replica.cc:330] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: stopping tablet replica
I20260809 05:21:18.219223 26070 raft_consensus.cc:2229] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:18.219694 26070 raft_consensus.cc:2258] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:18.221290 26070 ts_tablet_manager.cc:1905] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:18.225961 26073 ts_tablet_manager.cc:1918] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:18.226248 26073 log.cc:1196] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/f08c17d0a3e54390a3b2847f59ba8c97
I20260809 05:21:18.226253 26071 ts_tablet_manager.cc:1918] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:18.226411 26071 log.cc:1196] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/f08c17d0a3e54390a3b2847f59ba8c97
I20260809 05:21:18.226905 26073 ts_tablet_manager.cc:1939] T f08c17d0a3e54390a3b2847f59ba8c97 P 09299ab69a744b29bbd601a0ab09ee35: Deleting consensus metadata
I20260809 05:21:18.227011 26071 ts_tablet_manager.cc:1939] T f08c17d0a3e54390a3b2847f59ba8c97 P 07fa601ab397415e9397148239fa13f0: Deleting consensus metadata
I20260809 05:21:18.228183 25088 catalog_manager.cc:4841] TS 07fa601ab397415e9397148239fa13f0 (127.22.1.66:42135): tablet f08c17d0a3e54390a3b2847f59ba8c97 (table test-table [id=7ea9ccde4c57493e8869b5add4b8b365]) successfully deleted
I20260809 05:21:18.228185 25087 catalog_manager.cc:4841] TS 09299ab69a744b29bbd601a0ab09ee35 (127.22.1.65:41361): tablet f08c17d0a3e54390a3b2847f59ba8c97 (table test-table [id=7ea9ccde4c57493e8869b5add4b8b365]) successfully deleted
I20260809 05:21:18.229410 26070 ts_tablet_manager.cc:1918] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260809 05:21:18.229664 26070 log.cc:1196] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/f08c17d0a3e54390a3b2847f59ba8c97
I20260809 05:21:18.230270 26070 ts_tablet_manager.cc:1939] T f08c17d0a3e54390a3b2847f59ba8c97 P 90018db4012f4390bf8d71c901adeb36: Deleting consensus metadata
I20260809 05:21:18.231350 25087 catalog_manager.cc:4841] TS 90018db4012f4390bf8d71c901adeb36 (127.22.1.67:45263): tablet f08c17d0a3e54390a3b2847f59ba8c97 (table test-table [id=7ea9ccde4c57493e8869b5add4b8b365]) successfully deleted
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:18.253737 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:18.206107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.22.1.65:41361)
0809 05:21:18.224853 (+ 18746us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:18.224860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:18.224917 (+ 57us) server_negotiation.cc:407] Connection header received
0809 05:21:18.224986 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:18.224991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:18.225096 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:18.228357 (+ 3261us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:18.228514 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.230372 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.232065 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.232673 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.232777 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:18.246683 (+ 13906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:18.246704 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:18.246725 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:18.246806 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:18.250865 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.251697 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.251705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.251707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.251895 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.252262 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.252265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.252267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.252635 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:18.252867 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:18.253144 (+ 277us) server_negotiation.cc:299] Negotiation successful
0809 05:21:18.253361 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18540,"thread_start_us":19156,"threads_started":1}
W20260809 05:21:18.254565 25416 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41496
W20260809 05:21:18.255343 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25225: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:18.255635 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 25225
I20260809 05:21:18.289252 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25434
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:18.305215 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:18.290094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.22.1.66:42135)
0809 05:21:18.290821 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:18.290826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:18.290843 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:21:18.290904 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:18.290907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:18.290977 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:18.291077 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:18.292555 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.293506 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.294779 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.295185 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.295307 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:18.299428 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:18.299471 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:18.299490 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:18.299531 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:18.302572 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.303229 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.303236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.303238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.303373 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.303681 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.303684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.303686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.304064 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:18.304298 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:18.304628 (+ 330us) server_negotiation.cc:299] Negotiation successful
0809 05:21:18.304809 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":335,"threads_started":1}
W20260809 05:21:18.306070 25632 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33958
W20260809 05:21:18.306741 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25434: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:18.307039 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 25434
I20260809 05:21:18.335887 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25644
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[25047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252875, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:18.354631 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:18.336784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.22.1.67:45263)
0809 05:21:18.337759 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:18.337764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:18.337781 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:21:18.337837 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:18.337841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:18.337902 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:18.337998 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:18.339453 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.340629 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.342029 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:18.342680 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:18.342802 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:18.347410 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:18.347431 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:18.347454 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:18.347506 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:18.351441 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.352212 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.352222 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.352226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.352398 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:18.352887 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:18.352892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:18.352896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:18.353460 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:18.353763 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:18.353998 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 05:21:18.354241 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":730,"threads_started":1}
W20260809 05:21:18.355387 25831 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53500
W20260809 05:21:18.356072 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25644: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:18.356227 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 25644
I20260809 05:21:18.397037 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 25062
I20260809 05:21:18.398276 25173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:18.513688 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 25062
2026-08-09T05:21:18Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9045 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[26083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[26083](info): set up 2 sockets
Aug 09 05:21:18 dist-test-slave-rmhg krb5kdc[26083](info): commencing operation
krb5kdc: starting...
W20260809 05:21:19.702592 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:19 dist-test-slave-rmhg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:19.914997 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43687
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:20.248751 26102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:20.249075 26102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:20.249217 26102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:20.256439 26102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:20.256547 26102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:20.256601 26102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:20.256690 26102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:20.256748 26102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:20.266697 26102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:20.268081 26102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:20.269454 26102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:20.281435 26109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:20.281450 26110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:20.281507 26112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:20.282049 26102 server_base.cc:1034] running on GCE node
I20260809 05:21:20.282920 26102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:20.284467 26102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:20.285728 26102 hybrid_clock.cc:648] HybridClock initialized: now 1786252880285652 us; error 49 us; skew 500 ppm
I20260809 05:21:20.286104 26102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:20 dist-test-slave-rmhg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:20.289263 26102 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:20.290820 26102 webserver.cc:460] Webserver started at http://127.22.1.126:44631/ using document root <none> and password file <none>
I20260809 05:21:20.291515 26102 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:20.291635 26102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:20.291939 26102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:20.294937 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "b83597eb0c334ccab3ec060de985b74d"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "c1cb420b74c4508c083ca1eea3a95e70"
server_key_iv: "c2a0d6e83cc42e4981215a8c861ecf9e"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.295673 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "b83597eb0c334ccab3ec060de985b74d"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "c1cb420b74c4508c083ca1eea3a95e70"
server_key_iv: "c2a0d6e83cc42e4981215a8c861ecf9e"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.300891 26102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 05:21:20.304425 26119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:20.305876 26102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:20.306046 26102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "b83597eb0c334ccab3ec060de985b74d"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "c1cb420b74c4508c083ca1eea3a95e70"
server_key_iv: "c2a0d6e83cc42e4981215a8c861ecf9e"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.306233 26102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:20.326829 26102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:20.331398 26102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:20.331692 26102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:20.349967 26102 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:43687
I20260809 05:21:20.349997 26199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:43687 every 8 connection(s)
I20260809 05:21:20.351397 26102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:20.355294 26200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:20.356640 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 26102
I20260809 05:21:20.356863 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:20.357721 22533 external_mini_cluster.cc:1428] Setting key ebe168215eee7aa622168bc48983745a
I20260809 05:21:20.368716 26200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d: Bootstrap starting.
Aug 09 05:21:20 dist-test-slave-rmhg krb5kdc[26083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:20.372363 26200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:20.373667 26200 log.cc:824] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:20.377725 26200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d: No bootstrap required, opened a new log
I20260809 05:21:20.382674 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:20.360008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.22.1.126:43687)
0809 05:21:20.360809 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:20.360824 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:20.360868 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:21:20.362492 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:20.362519 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:20.362967 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:20.363350 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:20.364928 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:20.366366 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:20.367610 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:20.368296 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:20.368484 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:20.372798 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:20.372807 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:20.372827 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:20.372871 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:20.376646 (+ 3775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:20.377689 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:20.377699 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:20.377729 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:20.377968 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:20.378446 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:20.378452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:20.378455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:20.379275 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:20.380683 (+ 1408us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:20.381030 (+ 347us) server_negotiation.cc:299] Negotiation successful
0809 05:21:20.381559 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":346,"threads_started":1}
I20260809 05:21:20.387295 26200 raft_consensus.cc:348] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } }
I20260809 05:21:20.387669 26200 raft_consensus.cc:374] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:20.387775 26200 raft_consensus.cc:729] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b83597eb0c334ccab3ec060de985b74d, State: Initialized, Role: FOLLOWER
I20260809 05:21:20.388315 26200 consensus_queue.cc:260] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } }
I20260809 05:21:20.388546 26200 raft_consensus.cc:388] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:20.388674 26200 raft_consensus.cc:482] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:20.388818 26200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:20.391001 26200 raft_consensus.cc:504] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } }
I20260809 05:21:20.391499 26200 leader_election.cc:302] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b83597eb0c334ccab3ec060de985b74d; no voters:
I20260809 05:21:20.392135 26200 leader_election.cc:288] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:20.392261 26208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:20.393304 26208 raft_consensus.cc:686] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [term 1 LEADER]: Becoming Leader. State: Replica: b83597eb0c334ccab3ec060de985b74d, State: Running, Role: LEADER
I20260809 05:21:20.393879 26208 consensus_queue.cc:237] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } }
I20260809 05:21:20.394678 26200 sys_catalog.cc:564] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:20.396889 26213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b83597eb0c334ccab3ec060de985b74d. Latest consensus state: current_term: 1 leader_uuid: "b83597eb0c334ccab3ec060de985b74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } } }
I20260809 05:21:20.396967 26212 sys_catalog.cc:455] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b83597eb0c334ccab3ec060de985b74d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b83597eb0c334ccab3ec060de985b74d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43687 } } }
I20260809 05:21:20.397322 26213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:20.397351 26212 sys_catalog.cc:458] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:20.398241 26219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:20.404512 26219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:20.414707 26219 catalog_manager.cc:1347] Generated new cluster ID: 421a89f0b9d1428990f8307bfb8e772e
I20260809 05:21:20.414850 26219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:20.423861 26219 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:20.425921 26219 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:20.440135 26219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b83597eb0c334ccab3ec060de985b74d: Generated new TSK 0
I20260809 05:21:20.441119 26219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:20.508414 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:20.847154 26250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:20.847497 26250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:20.847637 26250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:20.855077 26250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:20.855226 26250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:20.855391 26250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:20.865422 26250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:20.866803 26250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:20.868211 26250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:20.881600 26259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:20.881765 26257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:20.881909 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:20.882966 26250 server_base.cc:1034] running on GCE node
I20260809 05:21:20.883734 26250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:20.884729 26250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:20.885941 26250 hybrid_clock.cc:648] HybridClock initialized: now 1786252880885877 us; error 46 us; skew 500 ppm
I20260809 05:21:20.886270 26250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:20 dist-test-slave-rmhg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:20.889439 26250 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:20.890952 26250 webserver.cc:460] Webserver started at http://127.22.1.65:38917/ using document root <none> and password file <none>
I20260809 05:21:20.891678 26250 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:20.891786 26250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:20.892107 26250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:20.895226 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "025a7d1d909d423f8d2c543be5fd97b6"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "15fc41ce914aa90489a2bcaca0059d49"
server_key_iv: "5cbf5f22be75fd939a5c0c4ff3fb2b1a"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.896019 26250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "025a7d1d909d423f8d2c543be5fd97b6"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "15fc41ce914aa90489a2bcaca0059d49"
server_key_iv: "5cbf5f22be75fd939a5c0c4ff3fb2b1a"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.901270 26250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 05:21:20.904747 26271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:20.906294 26250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:20.906451 26250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "025a7d1d909d423f8d2c543be5fd97b6"
format_stamp: "Formatted at 2026-08-09 05:21:20 on dist-test-slave-rmhg"
server_key: "15fc41ce914aa90489a2bcaca0059d49"
server_key_iv: "5cbf5f22be75fd939a5c0c4ff3fb2b1a"
server_key_version: "encryptionkey@0"
I20260809 05:21:20.906679 26250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:20.941663 26250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:20.948098 26250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:20.948449 26250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:20.951017 26250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:20.953002 26250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:20.953083 26250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:20.953203 26250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:20.953272 26250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260809 05:21:20.992239 26250 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:34611
I20260809 05:21:20.992319 26435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:34611 every 8 connection(s)
I20260809 05:21:20.993713 26250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:21.003654 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 26250
I20260809 05:21:21.003846 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:21.004663 22533 external_mini_cluster.cc:1428] Setting key 3fd66be4bb60832ea38896868a2fb763
Aug 09 05:21:21 dist-test-slave-rmhg krb5kdc[26083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252880, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:21.018028 26440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:20.997472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:35795 (local address 127.22.1.126:43687)
0809 05:21:20.997960 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:20.997966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:21.000742 (+ 2776us) server_negotiation.cc:407] Connection header received
0809 05:21:21.000827 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:21.000833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:21.000930 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:21.001027 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:21.002903 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.004146 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.006699 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.007266 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.007390 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:21.012028 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:21.012034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:21.012038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:21.012074 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:21.014724 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:21.015532 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:21.015537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:21.015540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:21.015670 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:21.016186 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:21.016190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:21.016193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:21.016479 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:21.016602 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:21.017559 (+ 957us) server_negotiation.cc:299] Negotiation successful
0809 05:21:21.017744 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":246,"threads_started":1}
I20260809 05:21:21.019539 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:20.997066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43687 (local address 127.22.1.65:35795)
0809 05:21:20.997976 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:20.998028 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:21.000364 (+ 2336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:21.001528 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:21.001537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:21.002096 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:21.002715 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:21.002734 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.004358 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.004364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:21.006546 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:21.006563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.007528 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.007536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:21.007862 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:21.008628 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:21.008661 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:21.011728 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:21.014891 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:21.014902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:21.014927 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:21.015385 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:21.015789 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:21.015794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:21.015798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:21.016062 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:21.016616 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:21.016625 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:21.017100 (+ 475us) client_negotiation.cc:769] Sending connection context
0809 05:21:21.017451 (+ 351us) client_negotiation.cc:240] Negotiation successful
0809 05:21:21.018025 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":383,"threads_started":1}
I20260809 05:21:21.021307 26437 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43687
I20260809 05:21:21.021649 26437 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:21.022408 26437 heartbeater.cc:511] Master 127.22.1.126:43687 requested a full tablet report, sending...
I20260809 05:21:21.024878 26154 ts_manager.cc:194] Registered new tserver with Master: 025a7d1d909d423f8d2c543be5fd97b6 (127.22.1.65:34611)
I20260809 05:21:21.026445 26154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:35795
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:21.055869 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:21.395040 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:21.395366 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:21.395506 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:21.402966 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:21.403076 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:21.403229 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:21.413384 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:21.414714 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:21.416042 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:21.429698 26453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:21.429749 26456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:21.429908 26452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:21.430795 26445 server_base.cc:1034] running on GCE node
I20260809 05:21:21.431411 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:21.432396 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:21.433622 26445 hybrid_clock.cc:648] HybridClock initialized: now 1786252881433571 us; error 37 us; skew 500 ppm
I20260809 05:21:21.433950 26445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:21 dist-test-slave-rmhg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252881, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:21.437073 26445 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:21.438589 26445 webserver.cc:460] Webserver started at http://127.22.1.66:36033/ using document root <none> and password file <none>
I20260809 05:21:21.439294 26445 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:21.439412 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:21.439761 26445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:21.442924 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "092d7cc795164a88853bb33f82d62aec"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "fea81b05101985153a431381e20d763e"
server_key_iv: "afbe8b29aa2b0fe3e596d2236558b1fb"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.443713 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "092d7cc795164a88853bb33f82d62aec"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "fea81b05101985153a431381e20d763e"
server_key_iv: "afbe8b29aa2b0fe3e596d2236558b1fb"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.448918 26445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:21.452414 26468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:21.453976 26445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:21.454222 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "092d7cc795164a88853bb33f82d62aec"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "fea81b05101985153a431381e20d763e"
server_key_iv: "afbe8b29aa2b0fe3e596d2236558b1fb"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.454434 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:21.480581 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:21.485380 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:21.485675 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:21.486714 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:21.488113 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:21.488191 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:21.488312 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:21.488368 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:21.520042 26445 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:34195
I20260809 05:21:21.520103 26640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:34195 every 8 connection(s)
I20260809 05:21:21.521467 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:21.527683 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 26445
I20260809 05:21:21.527870 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:21.528461 22533 external_mini_cluster.cc:1428] Setting key d482312f3a33af3f106939abc8275c14
Aug 09 05:21:21 dist-test-slave-rmhg krb5kdc[26083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252881, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:21.545511 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:21.524126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:47761 (local address 127.22.1.126:43687)
0809 05:21:21.524640 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:21.524647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:21.525613 (+ 966us) server_negotiation.cc:407] Connection header received
0809 05:21:21.527504 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:21.527509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:21.527580 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:21.527676 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:21.529862 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.530946 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.533477 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.533943 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.534067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:21.538499 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:21.538508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:21.538513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:21.538563 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:21.541971 (+ 3408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:21.542898 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:21.542904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:21.542907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:21.543070 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:21.543531 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:21.543536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:21.543539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:21.543915 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:21.544075 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:21.544899 (+ 824us) server_negotiation.cc:299] Negotiation successful
0809 05:21:21.545134 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":232,"threads_started":1}
I20260809 05:21:21.546517 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:21.524628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43687 (local address 127.22.1.66:47761)
0809 05:21:21.525406 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:21.525485 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:21.527156 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:21.527987 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:21.527998 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:21.528772 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:21.529635 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:21.529663 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.531156 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.531162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:21.533272 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:21.533298 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:21.534204 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:21.534211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:21.534533 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:21.535303 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:21.535333 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:21.538090 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:21.542159 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:21.542168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:21.542185 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:21.542728 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:21.543193 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:21.543196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:21.543199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:21.543404 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:21.544053 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:21.544060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:21.544456 (+ 396us) client_negotiation.cc:769] Sending connection context
0809 05:21:21.544740 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 05:21:21.545255 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":289,"threads_started":1}
I20260809 05:21:21.548257 26641 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43687
I20260809 05:21:21.548671 26641 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:21.549724 26641 heartbeater.cc:511] Master 127.22.1.126:43687 requested a full tablet report, sending...
I20260809 05:21:21.551532 26154 ts_manager.cc:194] Registered new tserver with Master: 092d7cc795164a88853bb33f82d62aec (127.22.1.66:34195)
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:21:21.552403 26154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:47761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:21.580914 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:21.913004 26650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:21.913358 26650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:21.913537 26650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:21.920876 26650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:21.921008 26650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:21.921195 26650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:21.931422 26650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:21.932767 26650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:21.934158 26650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:21.947243 26659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:21.947207 26662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:21.947170 26660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:21.948417 26650 server_base.cc:1034] running on GCE node
I20260809 05:21:21.949021 26650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:21.950035 26650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:21.951262 26650 hybrid_clock.cc:648] HybridClock initialized: now 1786252881951213 us; error 32 us; skew 500 ppm
I20260809 05:21:21.951576 26650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:21 dist-test-slave-rmhg krb5kdc[26083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252881, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:21.954725 26650 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:21.956285 26650 webserver.cc:460] Webserver started at http://127.22.1.67:38283/ using document root <none> and password file <none>
I20260809 05:21:21.957134 26650 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:21.957285 26650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:21.957615 26650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:21.960704 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "7e925299a54548dd8ca3ae594b5a68f5"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "5c91421f8b1a7398b73ff4bdfb15468b"
server_key_iv: "a6fc5775fbf1af71b46672dcf1b0daca"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.961562 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "7e925299a54548dd8ca3ae594b5a68f5"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "5c91421f8b1a7398b73ff4bdfb15468b"
server_key_iv: "a6fc5775fbf1af71b46672dcf1b0daca"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.966688 26650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:21:21.970229 26674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:21.971623 26650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:21.971781 26650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "7e925299a54548dd8ca3ae594b5a68f5"
format_stamp: "Formatted at 2026-08-09 05:21:21 on dist-test-slave-rmhg"
server_key: "5c91421f8b1a7398b73ff4bdfb15468b"
server_key_iv: "a6fc5775fbf1af71b46672dcf1b0daca"
server_key_version: "encryptionkey@0"
I20260809 05:21:21.971959 26650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:22.001487 26650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:22.005712 26650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:22.006008 26650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:22.007014 26650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:22.008404 26650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:22.008496 26650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:22.008615 26650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:22.008667 26650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:22.029907 26437 heartbeater.cc:503] Master 127.22.1.126:43687 was elected leader, sending a full tablet report...
I20260809 05:21:22.042161 26650 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:41593
I20260809 05:21:22.042233 26841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:41593 every 8 connection(s)
I20260809 05:21:22.043538 26650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:22.053301 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 26650
I20260809 05:21:22.053498 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:22.054075 22533 external_mini_cluster.cc:1428] Setting key 76bb6835a13059b29d15de97d13f6ca1
Aug 09 05:21:22 dist-test-slave-rmhg krb5kdc[26083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252881, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:22.063926 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:22.046310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:40237 (local address 127.22.1.126:43687)
0809 05:21:22.046797 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:22.046801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:22.047840 (+ 1039us) server_negotiation.cc:407] Connection header received
0809 05:21:22.049761 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:22.049766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:22.049836 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:22.049949 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:22.051495 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:22.052284 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:22.054377 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:22.054800 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:22.054905 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:22.058543 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:22.058548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:22.058552 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:22.058605 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:22.061279 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:22.061986 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:22.061991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:22.061993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:22.062120 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:22.062486 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:22.062489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:22.062491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:22.062743 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:22.062856 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:22.063490 (+ 634us) server_negotiation.cc:299] Negotiation successful
0809 05:21:22.063657 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
I20260809 05:21:22.064960 26847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:22.046851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43687 (local address 127.22.1.67:40237)
0809 05:21:22.047656 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:22.047710 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:22.049497 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:22.050196 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:22.050204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:22.050724 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:22.051325 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:22.051342 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:22.052401 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:22.052404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:22.054232 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:22.054248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:22.055036 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:22.055045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:22.055272 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:22.055982 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:22.056007 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:22.058338 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:22.061409 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:22.061417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:22.061432 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:22.061867 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:22.062205 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:22.062208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:22.062210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:22.062396 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:22.062841 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:22.062847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:22.063190 (+ 343us) client_negotiation.cc:769] Sending connection context
0809 05:21:22.063410 (+ 220us) client_negotiation.cc:240] Negotiation successful
0809 05:21:22.063812 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":325,"threads_started":1}
I20260809 05:21:22.066418 26844 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43687
I20260809 05:21:22.066742 26844 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:22.067528 26844 heartbeater.cc:511] Master 127.22.1.126:43687 requested a full tablet report, sending...
I20260809 05:21:22.069190 26154 ts_manager.cc:194] Registered new tserver with Master: 7e925299a54548dd8ca3ae594b5a68f5 (127.22.1.67:41593)
I20260809 05:21:22.069794 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:22.070125 26154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:40237
W20260809 05:21:22.102710 26846 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:22.100096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.22.1.126:43687)
0809 05:21:22.100272 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:22.100277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:22.100770 (+ 493us) server_negotiation.cc:407] Connection header received
0809 05:21:22.101029 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:22.101034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:22.101108 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:22.101251 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:22.102501 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53486: BlockingRecv error: recv got EOF from 127.0.0.1:53486 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:21:22.105901 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26250
W20260809 05:21:22.108793 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26250: Not authorized: Client connection negotiation failed: client connection to 127.22.1.65:34611: server requires authentication, but client does not have Kerberos credentials available
I20260809 05:21:22.109038 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 26250
I20260809 05:21:22.131251 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26445
W20260809 05:21:22.134119 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26445: Not authorized: Client connection negotiation failed: client connection to 127.22.1.66:34195: server requires authentication, but client does not have Kerberos credentials available
I20260809 05:21:22.134248 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 26445
I20260809 05:21:22.165354 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26650
W20260809 05:21:22.168399 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26650: Not authorized: Client connection negotiation failed: client connection to 127.22.1.67:41593: server requires authentication, but client does not have Kerberos credentials available
I20260809 05:21:22.168610 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 26650
I20260809 05:21:22.190280 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26102
I20260809 05:21:22.191622 26194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:22.295260 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 26102
2026-08-09T05:21:22Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3777 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:22 dist-test-slave-rmhg krb5kdc[26873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:22 dist-test-slave-rmhg krb5kdc[26873](info): set up 2 sockets
Aug 09 05:21:22 dist-test-slave-rmhg krb5kdc[26873](info): commencing operation
krb5kdc: starting...
W20260809 05:21:23.483440 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:23 dist-test-slave-rmhg krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:23.686929 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35323
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:24.015041 26889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:24.015342 26889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:24.015472 26889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:24.022603 26889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:24.022713 26889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:24.022765 26889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:24.022845 26889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:24.022908 26889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:24.032701 26889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:24.034008 26889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:24.035333 26889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:24.047156 26899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:24.047117 26900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:24.047080 26903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:24.047932 26889 server_base.cc:1034] running on GCE node
I20260809 05:21:24.048717 26889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:24.050175 26889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:24.051384 26889 hybrid_clock.cc:648] HybridClock initialized: now 1786252884051344 us; error 31 us; skew 500 ppm
I20260809 05:21:24.051726 26889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:24 dist-test-slave-rmhg krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252884, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:24.058735 26889 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:24.060237 26889 webserver.cc:460] Webserver started at http://127.22.1.126:44823/ using document root <none> and password file <none>
I20260809 05:21:24.060915 26889 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:24.061035 26889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:24.061389 26889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:24.064354 26889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "be0270629d62442f9c886cdbbdf000e1"
format_stamp: "Formatted at 2026-08-09 05:21:24 on dist-test-slave-rmhg"
server_key: "c5c3463c68d899378123098e78898af5"
server_key_iv: "9992061e57a9ca633454f698feff4709"
server_key_version: "encryptionkey@0"
I20260809 05:21:24.065200 26889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "be0270629d62442f9c886cdbbdf000e1"
format_stamp: "Formatted at 2026-08-09 05:21:24 on dist-test-slave-rmhg"
server_key: "c5c3463c68d899378123098e78898af5"
server_key_iv: "9992061e57a9ca633454f698feff4709"
server_key_version: "encryptionkey@0"
I20260809 05:21:24.070401 26889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:21:24.073866 26915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:24.075218 26889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:24.075376 26889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "be0270629d62442f9c886cdbbdf000e1"
format_stamp: "Formatted at 2026-08-09 05:21:24 on dist-test-slave-rmhg"
server_key: "c5c3463c68d899378123098e78898af5"
server_key_iv: "9992061e57a9ca633454f698feff4709"
server_key_version: "encryptionkey@0"
I20260809 05:21:24.075559 26889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:24.100303 26889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:24.104499 26889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:24.104801 26889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:24.122702 26889 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35323
I20260809 05:21:24.122740 26999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35323 every 8 connection(s)
I20260809 05:21:24.124051 26889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:24.127400 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 26889
I20260809 05:21:24.127641 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:24.128057 27000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:24.128288 22533 external_mini_cluster.cc:1428] Setting key efe96c1642f2b31dab0923a452a3a0df
I20260809 05:21:24.143013 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1: Bootstrap starting.
Aug 09 05:21:24 dist-test-slave-rmhg krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:24.147173 27000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:24.148599 27000 log.cc:824] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:24.152144 27000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1: No bootstrap required, opened a new log
I20260809 05:21:24.153205 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:24.130130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.22.1.126:35323)
0809 05:21:24.131048 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:24.131069 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:24.131122 (+ 53us) server_negotiation.cc:407] Connection header received
0809 05:21:24.133106 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:24.133139 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:24.133774 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:24.134281 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:24.135588 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:24.137695 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:24.139012 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:24.140150 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:24.140360 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:24.144408 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:24.144420 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:24.144462 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:24.144522 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:24.148324 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:24.149072 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:24.149083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:24.149093 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:24.149365 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:24.149786 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:24.149793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:24.149795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:24.150565 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:24.150847 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:24.151222 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 05:21:24.151688 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":354,"threads_started":1}
I20260809 05:21:24.157104 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:21:24.161824 27000 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } }
I20260809 05:21:24.162304 27000 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:24.162407 27000 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0270629d62442f9c886cdbbdf000e1, State: Initialized, Role: FOLLOWER
I20260809 05:21:24.163076 27000 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } }
I20260809 05:21:24.163337 27000 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:24.163468 27000 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:24.163630 27000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:24.166546 27000 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } }
I20260809 05:21:24.167101 27000 leader_election.cc:302] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0270629d62442f9c886cdbbdf000e1; no voters:
I20260809 05:21:24.167712 27000 leader_election.cc:288] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:24.167829 27007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:24.168952 27007 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [term 1 LEADER]: Becoming Leader. State: Replica: be0270629d62442f9c886cdbbdf000e1, State: Running, Role: LEADER
I20260809 05:21:24.169615 27007 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } }
I20260809 05:21:24.170146 27000 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:24.172389 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0270629d62442f9c886cdbbdf000e1. Latest consensus state: current_term: 1 leader_uuid: "be0270629d62442f9c886cdbbdf000e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } } }
I20260809 05:21:24.172744 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:24.173369 27008 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0270629d62442f9c886cdbbdf000e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0270629d62442f9c886cdbbdf000e1" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35323 } } }
I20260809 05:21:24.173676 27008 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:24.175879 27018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:24.184684 27018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:24.193269 27018 catalog_manager.cc:1347] Generated new cluster ID: 2125c30a4b2e4b84a868996a159eca14
I20260809 05:21:24.193454 27018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:24.204372 27018 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:24.206037 27018 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:24.208989 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:24.199999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38696 (local address 127.22.1.126:35323)
0809 05:21:24.200176 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:24.200182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:24.200766 (+ 584us) server_negotiation.cc:407] Connection header received
0809 05:21:24.200960 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:24.200965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:24.201060 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:24.201225 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:24.202456 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:24.203224 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:24.205097 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:24.205745 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:24.205933 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:24.207115 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:24.207121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:24.207126 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:24.208138 (+ 1012us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:24.208195 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:24.208222 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:24.208325 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:24.208488 (+ 163us) server_negotiation.cc:299] Negotiation successful
0809 05:21:24.208619 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":62}
I20260809 05:21:24.219969 27018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0270629d62442f9c886cdbbdf000e1: Generated new TSK 0
I20260809 05:21:24.221119 27018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:21:24.235080 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 26889
I20260809 05:21:24.236547 26991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:24.326102 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 26889
2026-08-09T05:21:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2023 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:24 dist-test-slave-rmhg krb5kdc[27044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:24 dist-test-slave-rmhg krb5kdc[27044](info): set up 2 sockets
Aug 09 05:21:24 dist-test-slave-rmhg krb5kdc[27044](info): commencing operation
krb5kdc: starting...
W20260809 05:21:25.497920 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:25 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:25.710839 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44365
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:26.038105 27067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:26.038411 27067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:26.038501 27067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:26.045679 27067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:26.045802 27067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:26.045855 27067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:26.045910 27067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:26.045954 27067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:26.055685 27067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:26.057049 27067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:26.058360 27067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:26.069846 27074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:26.069924 27073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:26.069888 27078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:26.071277 27067 server_base.cc:1034] running on GCE node
I20260809 05:21:26.072105 27067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:26.073597 27067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:26.074798 27067 hybrid_clock.cc:648] HybridClock initialized: now 1786252886074748 us; error 43 us; skew 500 ppm
I20260809 05:21:26.075133 27067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:26 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:26.078188 27067 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:26.079734 27067 webserver.cc:460] Webserver started at http://127.22.1.126:39579/ using document root <none> and password file <none>
I20260809 05:21:26.080413 27067 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:26.080541 27067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:26.080835 27067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:26.083884 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "fcb06fb794d445f790c3e480566bcd6a"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "248f441f1e5de2eb80494942bb79e8cc"
server_key_iv: "cf3d7f9cfc4bbd4c383eb9d643577842"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.084671 27067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "fcb06fb794d445f790c3e480566bcd6a"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "248f441f1e5de2eb80494942bb79e8cc"
server_key_iv: "cf3d7f9cfc4bbd4c383eb9d643577842"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.090041 27067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 05:21:26.093566 27087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:26.094993 27067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:26.095161 27067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "fcb06fb794d445f790c3e480566bcd6a"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "248f441f1e5de2eb80494942bb79e8cc"
server_key_iv: "cf3d7f9cfc4bbd4c383eb9d643577842"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.095345 27067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:26.126820 27067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:26.131304 27067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:26.131606 27067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:26.149322 27067 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:44365
I20260809 05:21:26.149343 27167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:44365 every 8 connection(s)
I20260809 05:21:26.150671 27067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:26.152232 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 27067
I20260809 05:21:26.152483 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:26.153318 22533 external_mini_cluster.cc:1428] Setting key 0ea56e353477c8c1aa6363689153c2e6
I20260809 05:21:26.155479 27168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 05:21:26 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:26.168902 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: Bootstrap starting.
I20260809 05:21:26.172258 27168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:26.173604 27168 log.cc:824] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:26.176687 27172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:26.155379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.22.1.126:44365)
0809 05:21:26.156144 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:26.156157 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:26.156199 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:21:26.157832 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:26.157856 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:26.158290 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:26.158647 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:26.160393 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.161831 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.163518 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.164167 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.164291 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:26.169178 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:26.169193 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:26.169213 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:26.169263 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:26.172126 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:26.172956 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:26.172963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:26.172970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:26.173145 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:26.173582 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:26.173585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:26.173590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:26.174202 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:26.174526 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:26.174910 (+ 384us) server_negotiation.cc:299] Negotiation successful
0809 05:21:26.175344 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":347,"threads_started":1}
I20260809 05:21:26.177094 27168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: No bootstrap required, opened a new log
I20260809 05:21:26.184780 27168 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } }
I20260809 05:21:26.185286 27168 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:26.185397 27168 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb06fb794d445f790c3e480566bcd6a, State: Initialized, Role: FOLLOWER
I20260809 05:21:26.186038 27168 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } }
I20260809 05:21:26.186317 27168 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:26.186452 27168 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:26.186630 27168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:26.188951 27168 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } }
I20260809 05:21:26.189445 27168 leader_election.cc:302] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcb06fb794d445f790c3e480566bcd6a; no voters:
I20260809 05:21:26.190099 27168 leader_election.cc:288] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:26.190245 27175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:26.191334 27175 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [term 1 LEADER]: Becoming Leader. State: Replica: fcb06fb794d445f790c3e480566bcd6a, State: Running, Role: LEADER
I20260809 05:21:26.191946 27168 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:26.191844 27175 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } }
I20260809 05:21:26.197700 27176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcb06fb794d445f790c3e480566bcd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } } }
I20260809 05:21:26.198709 27176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:26.198132 27177 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb06fb794d445f790c3e480566bcd6a. Latest consensus state: current_term: 1 leader_uuid: "fcb06fb794d445f790c3e480566bcd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb06fb794d445f790c3e480566bcd6a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44365 } } }
I20260809 05:21:26.199023 27177 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a [sys.catalog]: This master's current role is: LEADER
W20260809 05:21:26.203236 27193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:21:26.203344 27193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:21:26.203467 27195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:26.205054 27195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:26.212495 27195 catalog_manager.cc:1347] Generated new cluster ID: 61dd1f40ede348e782cd0e58364a5134
I20260809 05:21:26.212605 27195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:26.227615 27195 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:26.228827 27195 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:26.236492 27195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcb06fb794d445f790c3e480566bcd6a: Generated new TSK 0
I20260809 05:21:26.237496 27195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:26.297469 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:26.644179 27202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:26.644495 27202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:26.644627 27202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:26.651993 27202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:26.652122 27202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:26.652285 27202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:26.662312 27202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:26.663681 27202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:26.665041 27202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:26.678500 27208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:26.678519 27211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:26.678722 27209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:26.679414 27202 server_base.cc:1034] running on GCE node
I20260809 05:21:26.679988 27202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:26.680962 27202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:26.682155 27202 hybrid_clock.cc:648] HybridClock initialized: now 1786252886682098 us; error 53 us; skew 500 ppm
I20260809 05:21:26.682466 27202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:26 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:26.685590 27202 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:26.687129 27202 webserver.cc:460] Webserver started at http://127.22.1.65:36595/ using document root <none> and password file <none>
I20260809 05:21:26.687845 27202 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:26.687973 27202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:26.688277 27202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:26.691332 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "d7dad39ea37e48a392c405f0e69d75ae"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "cbafe7013c5e5060e46e880337d16ed9"
server_key_iv: "14d734c414f1c6c0b1b1c22295393942"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.692201 27202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "d7dad39ea37e48a392c405f0e69d75ae"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "cbafe7013c5e5060e46e880337d16ed9"
server_key_iv: "14d734c414f1c6c0b1b1c22295393942"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.697803 27202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:26.701463 27220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:26.702823 27202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:26.702989 27202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "d7dad39ea37e48a392c405f0e69d75ae"
format_stamp: "Formatted at 2026-08-09 05:21:26 on dist-test-slave-rmhg"
server_key: "cbafe7013c5e5060e46e880337d16ed9"
server_key_iv: "14d734c414f1c6c0b1b1c22295393942"
server_key_version: "encryptionkey@0"
I20260809 05:21:26.703184 27202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:26.720611 27202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:26.724650 27202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:26.724939 27202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:26.726014 27202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:26.727439 27202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:26.727522 27202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:26.727654 27202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:26.727708 27202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:26.758684 27202 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:35325
I20260809 05:21:26.758731 27394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:35325 every 8 connection(s)
I20260809 05:21:26.760047 27202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:26.761109 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 27202
I20260809 05:21:26.761373 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:26.762251 22533 external_mini_cluster.cc:1428] Setting key e185cd2b16747a4ace44a2291dfb44f3
Aug 09 05:21:26 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252886, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:26.783654 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:26.763170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:50225 (local address 127.22.1.126:44365)
0809 05:21:26.763654 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:26.763660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:26.764816 (+ 1156us) server_negotiation.cc:407] Connection header received
0809 05:21:26.767344 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:26.767349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:26.767418 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:26.767520 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:26.769593 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.770313 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.772465 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.773064 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.773235 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:26.776769 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:26.776775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:26.776779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:26.776819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:26.780044 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:26.781007 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:26.781013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:26.781017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:26.781203 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:26.781671 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:26.781676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:26.781680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:26.782083 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:26.782253 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:26.783017 (+ 764us) server_negotiation.cc:299] Negotiation successful
0809 05:21:26.783260 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260809 05:21:26.784456 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:26.763768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44365 (local address 127.22.1.65:50225)
0809 05:21:26.764625 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:26.764689 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:26.767031 (+ 2342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:26.767799 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:26.767810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:26.768543 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:26.769388 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:26.769412 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.770486 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.770492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:26.772301 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:26.772315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:26.773365 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:26.773372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:26.773592 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:26.774268 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:26.774296 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:26.776508 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:26.780222 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:26.780230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:26.780246 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:26.780845 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:26.781305 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:26.781308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:26.781310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:26.781546 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:26.782209 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:26.782215 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:26.782601 (+ 386us) client_negotiation.cc:769] Sending connection context
0809 05:21:26.782843 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 05:21:26.783268 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
I20260809 05:21:26.786080 27395 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44365
I20260809 05:21:26.786401 27395 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:26.787242 27395 heartbeater.cc:511] Master 127.22.1.126:44365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
I20260809 05:21:26.789532 27114 ts_manager.cc:194] Registered new tserver with Master: d7dad39ea37e48a392c405f0e69d75ae (127.22.1.65:35325)
I20260809 05:21:26.791006 27114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:50225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:26.814177 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:27.138419 27404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:27.138712 27404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:27.138834 27404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:27.146031 27404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:27.146150 27404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:27.146304 27404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:27.155972 27404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:27.157248 27404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:27.158555 27404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:27.171343 27415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:27.171407 27413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:27.171398 27417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:27.172214 27404 server_base.cc:1034] running on GCE node
I20260809 05:21:27.172817 27404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:27.173862 27404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:27.175094 27404 hybrid_clock.cc:648] HybridClock initialized: now 1786252887175039 us; error 42 us; skew 500 ppm
I20260809 05:21:27.175418 27404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:27.178539 27404 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:27.179895 27404 webserver.cc:460] Webserver started at http://127.22.1.66:46601/ using document root <none> and password file <none>
I20260809 05:21:27.180576 27404 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:27.180696 27404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:27.180994 27404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:27.183933 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "651394a02cbb46c2b36091e61499a23a"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "669e1a073ebdfec0cbae9275c3f1e448"
server_key_iv: "336edba2ab32b18fd35e752e4f5b398e"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.184700 27404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "651394a02cbb46c2b36091e61499a23a"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "669e1a073ebdfec0cbae9275c3f1e448"
server_key_iv: "336edba2ab32b18fd35e752e4f5b398e"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.190397 27404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:21:27.194232 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.195818 27404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:27.195991 27404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "651394a02cbb46c2b36091e61499a23a"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "669e1a073ebdfec0cbae9275c3f1e448"
server_key_iv: "336edba2ab32b18fd35e752e4f5b398e"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.196192 27404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:27.232053 27404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:27.236636 27404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:27.236979 27404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:27.238026 27404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:27.239414 27404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:27.239494 27404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.239591 27404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:27.239635 27404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.270539 27404 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:41967
I20260809 05:21:27.270587 27613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:41967 every 8 connection(s)
I20260809 05:21:27.271803 27404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:27.275292 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 27404
I20260809 05:21:27.275483 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:27.275982 22533 external_mini_cluster.cc:1428] Setting key 4cb4302d1497d4eae184b85fe9dbce62
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:27.292323 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.274354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49069 (local address 127.22.1.126:44365)
0809 05:21:27.274544 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.274549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.275854 (+ 1305us) server_negotiation.cc:407] Connection header received
0809 05:21:27.277885 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.277890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.277965 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.278076 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.279572 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.280545 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.282159 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.282712 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.282833 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.286413 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.286418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.286422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.286456 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.288988 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.289861 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.289866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.289868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.289991 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.290455 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.290458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.290460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.290732 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.290840 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.291829 (+ 989us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.292044 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:21:27.293793 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.274858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44365 (local address 127.22.1.66:49069)
0809 05:21:27.275664 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:27.275720 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:27.277610 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:27.278278 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:27.278286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:27.278804 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:27.279383 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:27.279401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.280693 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.280697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:27.282016 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:27.282029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.282902 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.282907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:27.283114 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.283742 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:27.283769 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:27.286188 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:27.289130 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:27.289139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:27.289188 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:27.289746 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:27.290093 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:27.290097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:27.290100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:27.290351 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:27.290865 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:27.290872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:27.291311 (+ 439us) client_negotiation.cc:769] Sending connection context
0809 05:21:27.291658 (+ 347us) client_negotiation.cc:240] Negotiation successful
0809 05:21:27.292266 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260809 05:21:27.295413 27614 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44365
I20260809 05:21:27.295725 27614 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:27.296499 27614 heartbeater.cc:511] Master 127.22.1.126:44365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:21:27.298053 27114 ts_manager.cc:194] Registered new tserver with Master: 651394a02cbb46c2b36091e61499a23a (127.22.1.66:41967)
I20260809 05:21:27.298970 27114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:27.324512 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:27.650295 27620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:27.650614 27620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:27.650746 27620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:27.657992 27620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:27.658138 27620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:27.658295 27620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:27.668120 27620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:27.669492 27620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:27.670938 27620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:27.683653 27629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:27.683625 27631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:27.683689 27628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:27.684456 27620 server_base.cc:1034] running on GCE node
I20260809 05:21:27.685077 27620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:27.686039 27620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:27.687237 27620 hybrid_clock.cc:648] HybridClock initialized: now 1786252887687189 us; error 43 us; skew 500 ppm
I20260809 05:21:27.687546 27620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:27.690640 27620 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:27.692094 27620 webserver.cc:460] Webserver started at http://127.22.1.67:42027/ using document root <none> and password file <none>
I20260809 05:21:27.692797 27620 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:27.692904 27620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:27.693231 27620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:27.696228 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "4beff02c4f604a4593c6dfca66e56ed9"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "f6c8cd4ac8d1a82fd2019f4a412ba541"
server_key_iv: "f75e41e299aa9241a1dc17bc0d360ea7"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.697002 27620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "4beff02c4f604a4593c6dfca66e56ed9"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "f6c8cd4ac8d1a82fd2019f4a412ba541"
server_key_iv: "f75e41e299aa9241a1dc17bc0d360ea7"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.702533 27620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:21:27.706283 27642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.707829 27620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:27.707998 27620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "4beff02c4f604a4593c6dfca66e56ed9"
format_stamp: "Formatted at 2026-08-09 05:21:27 on dist-test-slave-rmhg"
server_key: "f6c8cd4ac8d1a82fd2019f4a412ba541"
server_key_iv: "f75e41e299aa9241a1dc17bc0d360ea7"
server_key_version: "encryptionkey@0"
I20260809 05:21:27.708195 27620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:27.723855 27620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:27.728137 27620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:27.728421 27620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:27.729485 27620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:27.730871 27620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:27.730942 27620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.731045 27620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:27.731098 27620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:27.762697 27620 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:37031
I20260809 05:21:27.762730 27815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:37031 every 8 connection(s)
I20260809 05:21:27.763979 27620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:27.764971 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 27620
I20260809 05:21:27.765198 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:27.765687 22533 external_mini_cluster.cc:1428] Setting key dce2e760e2fb8205f82bb5606b018f6b
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:27.784395 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.766836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:58173 (local address 127.22.1.126:44365)
0809 05:21:27.767022 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.767028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.768273 (+ 1245us) server_negotiation.cc:407] Connection header received
0809 05:21:27.770272 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.770278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.770383 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.770480 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.771949 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.772699 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.774346 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.774766 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.774876 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.778291 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.778296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.778300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.778333 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.781413 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.782193 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.782197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.782199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.782340 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.782762 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.782766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.782768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.783051 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.783162 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.783953 (+ 791us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.784143 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:21:27.785449 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.767327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44365 (local address 127.22.1.67:58173)
0809 05:21:27.768136 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:27.768174 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:27.770018 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:27.770724 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:27.770732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:27.771226 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:27.771789 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:27.771806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.772847 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.772852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:27.774206 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:27.774220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.774993 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.775001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:27.775220 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.775853 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:27.775875 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:27.778085 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:27.781581 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:27.781589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:27.781603 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:27.782069 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:27.782421 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:27.782424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:27.782426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:27.782674 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:27.783195 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:27.783202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:27.783576 (+ 374us) client_negotiation.cc:769] Sending connection context
0809 05:21:27.783821 (+ 245us) client_negotiation.cc:240] Negotiation successful
0809 05:21:27.784250 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":333,"threads_started":1}
I20260809 05:21:27.786933 27816 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44365
I20260809 05:21:27.787209 27816 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:27.787947 27816 heartbeater.cc:511] Master 127.22.1.126:44365 requested a full tablet report, sending...
I20260809 05:21:27.789358 27114 ts_manager.cc:194] Registered new tserver with Master: 4beff02c4f604a4593c6dfca66e56ed9 (127.22.1.67:37031)
I20260809 05:21:27.790160 27114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:58173
I20260809 05:21:27.794075 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:27.794821 27395 heartbeater.cc:503] Master 127.22.1.126:44365 was elected leader, sending a full tablet report...
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:27.844303 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.831414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.22.1.126:44365)
0809 05:21:27.831583 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.831588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.831973 (+ 385us) server_negotiation.cc:407] Connection header received
0809 05:21:27.832135 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.832139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.832217 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.832311 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.833662 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.834396 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.835706 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.836124 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.836223 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.839780 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.839785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.839789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.839821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.842209 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.842795 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.842798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.842800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.842926 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.843215 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.843218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.843220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.843522 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.843708 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.843904 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.844070 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260809 05:21:27.844897 27114 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41492
I20260809 05:21:27.846266 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27202
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:27.861240 27840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.847335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.22.1.65:35325)
0809 05:21:27.847916 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.847922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.847947 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:21:27.848001 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.848006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.848210 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.848331 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.849562 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.850556 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.851688 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.852233 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.852325 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.856061 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.856073 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.856090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.856124 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.858743 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.859348 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.859355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.859357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.859472 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.859836 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.859839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.859841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.860189 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.860406 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.860658 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.860854 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":303,"threads_started":1}
W20260809 05:21:27.862567 27388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42642
W20260809 05:21:27.863698 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:27.863880 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 27202
I20260809 05:21:27.886117 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27404
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:27.902261 27841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.887082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.22.1.66:41967)
0809 05:21:27.887656 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.887663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.887690 (+ 27us) server_negotiation.cc:407] Connection header received
0809 05:21:27.887751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.887757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.887992 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.888128 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.889361 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.890555 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.891826 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.892455 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.892558 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.896752 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.896765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.896782 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.896825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.899612 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.900315 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.900323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.900325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.900460 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.900764 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.900767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.900769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.901103 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.901488 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.901680 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.901877 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":298,"threads_started":1}
W20260809 05:21:27.903492 27609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57848
W20260809 05:21:27.904515 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27404: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:27.904678 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 27404
I20260809 05:21:27.925846 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27620
Aug 09 05:21:27 dist-test-slave-rmhg krb5kdc[27044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252887, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:27.942351 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:27.926874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.22.1.67:37031)
0809 05:21:27.927552 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:27.927560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:27.927589 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:21:27.927651 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:27.927657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:27.927866 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:27.927997 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:27.929400 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.930469 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.931726 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:27.932293 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:27.932384 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:27.936795 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:27.936808 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:27.936826 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:27.936863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:27.939457 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.940150 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.940159 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.940161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.940299 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:27.940700 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:27.940703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:27.940705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:27.941090 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:27.941536 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:27.941755 (+ 219us) server_negotiation.cc:299] Negotiation successful
0809 05:21:27.941990 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":362,"threads_started":1}
W20260809 05:21:27.943657 27810 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36002
W20260809 05:21:27.944763 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27620: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:21:27.944903 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 27620
I20260809 05:21:27.963617 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27067
I20260809 05:21:27.964820 27162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:28.066905 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 27067
2026-08-09T05:21:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3747 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:28 dist-test-slave-rmhg krb5kdc[27847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:28 dist-test-slave-rmhg krb5kdc[27847](info): set up 2 sockets
Aug 09 05:21:28 dist-test-slave-rmhg krb5kdc[27847](info): commencing operation
krb5kdc: starting...
W20260809 05:21:29.264674 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:29 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:29.488157 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38405
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:29.819511 27874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:29.819839 27874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:29.819968 27874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:29.827364 27874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:29.827491 27874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:29.827553 27874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:29.827643 27874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:29.827702 27874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:29.837809 27874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:29.839327 27874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:29.840754 27874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:29.853066 27883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:29.853034 27886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:29.853138 27894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:29.853767 27874 server_base.cc:1034] running on GCE node
I20260809 05:21:29.854592 27874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:29.856025 27874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:29.857249 27874 hybrid_clock.cc:648] HybridClock initialized: now 1786252889857189 us; error 47 us; skew 500 ppm
I20260809 05:21:29.857577 27874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:29 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252889, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:29.863428 27874 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:29.865041 27874 webserver.cc:460] Webserver started at http://127.22.1.126:45651/ using document root <none> and password file <none>
I20260809 05:21:29.865772 27874 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:29.865890 27874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:29.866217 27874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:29.869410 27874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "e11fc324fcc245019ffdcdfc84e68cff"
format_stamp: "Formatted at 2026-08-09 05:21:29 on dist-test-slave-rmhg"
server_key: "bcfdf50aecdaf0145becd29393a6f597"
server_key_iv: "a8ce837e9f72e9948225f9693312cf62"
server_key_version: "encryptionkey@0"
I20260809 05:21:29.870218 27874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "e11fc324fcc245019ffdcdfc84e68cff"
format_stamp: "Formatted at 2026-08-09 05:21:29 on dist-test-slave-rmhg"
server_key: "bcfdf50aecdaf0145becd29393a6f597"
server_key_iv: "a8ce837e9f72e9948225f9693312cf62"
server_key_version: "encryptionkey@0"
I20260809 05:21:29.875692 27874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 05:21:29.879451 27903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:29.880955 27874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:21:29.881181 27874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "e11fc324fcc245019ffdcdfc84e68cff"
format_stamp: "Formatted at 2026-08-09 05:21:29 on dist-test-slave-rmhg"
server_key: "bcfdf50aecdaf0145becd29393a6f597"
server_key_iv: "a8ce837e9f72e9948225f9693312cf62"
server_key_version: "encryptionkey@0"
I20260809 05:21:29.881397 27874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:29.905676 27874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:29.910081 27874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:29.910363 27874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:29.928058 27874 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38405
I20260809 05:21:29.928093 27982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38405 every 8 connection(s)
I20260809 05:21:29.929502 27874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:29.930490 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 27874
I20260809 05:21:29.930701 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:29.931353 22533 external_mini_cluster.cc:1428] Setting key 96d7df20c6f0da3e71c6f8b9b98cdfbd
I20260809 05:21:29.933869 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 05:21:29 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:29.946825 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff: Bootstrap starting.
I20260809 05:21:29.951143 27983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:29.952345 27983 log.cc:824] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:29.955813 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff: No bootstrap required, opened a new log
I20260809 05:21:29.955844 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:29.933402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.22.1.126:38405)
0809 05:21:29.934197 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:29.934209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:29.934251 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:21:29.935912 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:29.935937 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:29.936360 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:29.936725 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:29.938562 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:29.940028 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:29.941742 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:29.942414 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:29.942550 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:29.947218 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:29.947229 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:29.947252 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:29.947307 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:29.951194 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:29.951940 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:29.951949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:29.951957 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:29.952196 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:29.952530 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:29.952535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:29.952538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:29.953309 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:29.953543 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:29.954049 (+ 506us) server_negotiation.cc:299] Negotiation successful
0809 05:21:29.954477 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":373,"threads_started":1}
I20260809 05:21:29.963654 27983 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } }
I20260809 05:21:29.964008 27983 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:29.964099 27983 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11fc324fcc245019ffdcdfc84e68cff, State: Initialized, Role: FOLLOWER
I20260809 05:21:29.964708 27983 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } }
I20260809 05:21:29.964926 27983 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:29.965067 27983 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:29.965260 27983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:29.967351 27983 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } }
I20260809 05:21:29.967823 27983 leader_election.cc:302] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11fc324fcc245019ffdcdfc84e68cff; no voters:
I20260809 05:21:29.968381 27983 leader_election.cc:288] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:29.968523 27991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:29.969133 27991 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [term 1 LEADER]: Becoming Leader. State: Replica: e11fc324fcc245019ffdcdfc84e68cff, State: Running, Role: LEADER
I20260809 05:21:29.969693 27991 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } }
I20260809 05:21:29.970557 27983 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:29.972404 27994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11fc324fcc245019ffdcdfc84e68cff. Latest consensus state: current_term: 1 leader_uuid: "e11fc324fcc245019ffdcdfc84e68cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } } }
I20260809 05:21:29.972744 27994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:29.973631 28001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:29.973461 27992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11fc324fcc245019ffdcdfc84e68cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11fc324fcc245019ffdcdfc84e68cff" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38405 } } }
I20260809 05:21:29.973776 27992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:29.979194 28001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:29.989208 28001 catalog_manager.cc:1347] Generated new cluster ID: 7e2b000e185144af96316611cf559ce2
I20260809 05:21:29.989344 28001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:30.006105 28001 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:30.007443 28001 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:30.017737 28001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11fc324fcc245019ffdcdfc84e68cff: Generated new TSK 0
I20260809 05:21:30.018576 28001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:30.076764 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:30.441645 28026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:30.441944 28026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:30.442037 28026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:30.449458 28026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:30.449584 28026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:30.449707 28026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:30.459971 28026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:30.461354 28026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:30.462672 28026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:30.476090 28035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:30.476116 28036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:30.476146 28038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:30.476861 28026 server_base.cc:1034] running on GCE node
I20260809 05:21:30.477545 28026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:30.478555 28026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:30.479754 28026 hybrid_clock.cc:648] HybridClock initialized: now 1786252890479694 us; error 56 us; skew 500 ppm
I20260809 05:21:30.480082 28026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:30 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252890, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:30.483274 28026 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:30.484669 28026 webserver.cc:460] Webserver started at http://127.22.1.65:39871/ using document root <none> and password file <none>
I20260809 05:21:30.485453 28026 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:30.485559 28026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:30.485828 28026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:30.488783 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "595f56f1f21d4d8296983ba1b7e8bbfb"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "e3fc863895e6a3efd4d2f2f8e3655f5b"
server_key_iv: "77b3011b8b8e892260baa580371e5b0f"
server_key_version: "encryptionkey@0"
I20260809 05:21:30.489609 28026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "595f56f1f21d4d8296983ba1b7e8bbfb"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "e3fc863895e6a3efd4d2f2f8e3655f5b"
server_key_iv: "77b3011b8b8e892260baa580371e5b0f"
server_key_version: "encryptionkey@0"
I20260809 05:21:30.495172 28026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:30.498984 28049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:30.500511 28026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:21:30.500672 28026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "595f56f1f21d4d8296983ba1b7e8bbfb"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "e3fc863895e6a3efd4d2f2f8e3655f5b"
server_key_iv: "77b3011b8b8e892260baa580371e5b0f"
server_key_version: "encryptionkey@0"
I20260809 05:21:30.500856 28026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:30.512818 28026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:30.517402 28026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:30.517673 28026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:30.518646 28026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:30.520125 28026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:30.520205 28026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:30.520290 28026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:30.520334 28026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:30.551676 28026 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:36451
I20260809 05:21:30.551726 28225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:36451 every 8 connection(s)
I20260809 05:21:30.553031 28026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:30.562088 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 28026
I20260809 05:21:30.562314 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:30.563136 22533 external_mini_cluster.cc:1428] Setting key c9d6ac12bfcc89c5fef8d8d2c94f7571
Aug 09 05:21:30 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252890, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:30.574240 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:30.555620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:43557 (local address 127.22.1.126:38405)
0809 05:21:30.556042 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:30.556054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:30.556944 (+ 890us) server_negotiation.cc:407] Connection header received
0809 05:21:30.558829 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:30.558836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:30.558923 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:30.559021 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:30.560595 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:30.561375 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:30.563735 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:30.564188 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:30.564299 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:30.568143 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:30.568149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:30.568152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:30.568185 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:30.570995 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:30.571827 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:30.571833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:30.571836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:30.571963 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:30.572397 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:30.572402 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:30.572405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:30.572689 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:30.572824 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:30.573639 (+ 815us) server_negotiation.cc:299] Negotiation successful
0809 05:21:30.573877 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":223,"threads_started":1}
I20260809 05:21:30.575311 28229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:30.556056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38405 (local address 127.22.1.65:43557)
0809 05:21:30.556773 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:30.556819 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:30.558532 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:30.559277 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:30.559286 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:30.559830 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:30.560417 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:30.560434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:30.561541 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:30.561555 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:30.563574 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:30.563599 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:30.564426 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:30.564448 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:30.564755 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:30.565516 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:30.565542 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:30.567906 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:30.571154 (+ 3248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:30.571165 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:30.571183 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:30.571699 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:30.572080 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:30.572085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:30.572088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:30.572284 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:30.572818 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:30.572826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:30.573239 (+ 413us) client_negotiation.cc:769] Sending connection context
0809 05:21:30.573510 (+ 271us) client_negotiation.cc:240] Negotiation successful
0809 05:21:30.574090 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":274,"threads_started":1}
I20260809 05:21:30.577016 28226 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38405
I20260809 05:21:30.577405 28226 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:30.578400 28226 heartbeater.cc:511] Master 127.22.1.126:38405 requested a full tablet report, sending...
I20260809 05:21:30.581012 27930 ts_manager.cc:194] Registered new tserver with Master: 595f56f1f21d4d8296983ba1b7e8bbfb (127.22.1.65:36451)
I20260809 05:21:30.583132 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:43557
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:30.612515 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:30.945083 28232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:30.945506 28232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:30.945673 28232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:30.954429 28232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:30.954556 28232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:30.954672 28232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:30.964511 28232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:30.965821 28232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:30.967108 28232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:30.979974 28244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:30.980077 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:30.980057 28241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:30.980700 28232 server_base.cc:1034] running on GCE node
I20260809 05:21:30.981382 28232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:30.982332 28232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:30.983574 28232 hybrid_clock.cc:648] HybridClock initialized: now 1786252890983503 us; error 52 us; skew 500 ppm
I20260809 05:21:30.983891 28232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:30 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252890, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:30.987154 28232 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:30.988596 28232 webserver.cc:460] Webserver started at http://127.22.1.66:43521/ using document root <none> and password file <none>
I20260809 05:21:30.989344 28232 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:30.989468 28232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:30.989777 28232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:30.992745 28232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "4f12fcf6c0644df9a3e907b8e21f3d53"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "27240530ed4113c72e1fb9fff491ef03"
server_key_iv: "50541fd18d17b3667ea64e00f5a8c890"
server_key_version: "encryptionkey@0"
I20260809 05:21:30.993600 28232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "4f12fcf6c0644df9a3e907b8e21f3d53"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "27240530ed4113c72e1fb9fff491ef03"
server_key_iv: "50541fd18d17b3667ea64e00f5a8c890"
server_key_version: "encryptionkey@0"
I20260809 05:21:30.998817 28232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:21:31.002321 28256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:31.003755 28232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:31.003922 28232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "4f12fcf6c0644df9a3e907b8e21f3d53"
format_stamp: "Formatted at 2026-08-09 05:21:30 on dist-test-slave-rmhg"
server_key: "27240530ed4113c72e1fb9fff491ef03"
server_key_iv: "50541fd18d17b3667ea64e00f5a8c890"
server_key_version: "encryptionkey@0"
I20260809 05:21:31.004127 28232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:31.019055 28232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:31.024844 28232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:31.025218 28232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:31.026445 28232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:31.031333 28232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:31.031507 28232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 05:21:31.031742 28232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:31.031853 28232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:31.065122 28232 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:39879
I20260809 05:21:31.065248 28419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:39879 every 8 connection(s)
I20260809 05:21:31.066490 28232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:31.074455 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 28232
I20260809 05:21:31.074637 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:31.075145 22533 external_mini_cluster.cc:1428] Setting key 0d0e2f1ac76b39ed043593d5debbc529
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252890, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:31.088758 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.069653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49065 (local address 127.22.1.126:38405)
0809 05:21:31.069859 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.069865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.071289 (+ 1424us) server_negotiation.cc:407] Connection header received
0809 05:21:31.073973 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.073979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.074079 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.074194 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.076068 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.076922 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.078692 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.079137 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.079251 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.082786 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.082791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.082795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.082829 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.085565 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.086356 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.086360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.086363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.086497 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.086941 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.086945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.086947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.087240 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.087375 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.088175 (+ 800us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.088415 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:21:31.089867 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.070188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38405 (local address 127.22.1.66:49065)
0809 05:21:31.071083 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:31.071157 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:31.073675 (+ 2518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:31.074468 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:31.074482 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:31.075263 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:31.075882 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:31.075906 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.077078 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.077084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:31.078554 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:31.078576 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.079385 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.079394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:31.079671 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.080331 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:31.080355 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:31.082550 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:31.085711 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:31.085721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:31.085741 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:31.086221 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:31.086610 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:31.086616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:31.086620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:31.086820 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:31.087370 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:31.087378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:31.087762 (+ 384us) client_negotiation.cc:769] Sending connection context
0809 05:21:31.088014 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:21:31.088595 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":352,"threads_started":1}
I20260809 05:21:31.091378 28422 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38405
I20260809 05:21:31.091775 28422 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:31.092748 28422 heartbeater.cc:511] Master 127.22.1.126:38405 requested a full tablet report, sending...
I20260809 05:21:31.094552 27930 ts_manager.cc:194] Registered new tserver with Master: 4f12fcf6c0644df9a3e907b8e21f3d53 (127.22.1.66:39879)
I20260809 05:21:31.095758 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49065
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:31.125442 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:31.458256 28427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:31.458560 28427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:31.458702 28427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:31.465982 28427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:31.466109 28427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:31.466281 28427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:31.476176 28427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:31.477560 28427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:31.478986 28427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:31.492295 28434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:31.492274 28436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:31.492348 28438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:31.493343 28427 server_base.cc:1034] running on GCE node
I20260809 05:21:31.493990 28427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:31.495015 28427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:31.496292 28427 hybrid_clock.cc:648] HybridClock initialized: now 1786252891496235 us; error 42 us; skew 500 ppm
I20260809 05:21:31.496629 28427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:31.499840 28427 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:31.501384 28427 webserver.cc:460] Webserver started at http://127.22.1.67:38113/ using document root <none> and password file <none>
I20260809 05:21:31.502182 28427 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:31.502295 28427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:31.502585 28427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:31.505812 28427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "c1db7285f5c144648514dbdb877b358a"
format_stamp: "Formatted at 2026-08-09 05:21:31 on dist-test-slave-rmhg"
server_key: "b7bffda819c01f358c1e8f7cc571e5a8"
server_key_iv: "9ea9c8efb7ceb4c84dcbc776f08c3fc1"
server_key_version: "encryptionkey@0"
I20260809 05:21:31.506609 28427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "c1db7285f5c144648514dbdb877b358a"
format_stamp: "Formatted at 2026-08-09 05:21:31 on dist-test-slave-rmhg"
server_key: "b7bffda819c01f358c1e8f7cc571e5a8"
server_key_iv: "9ea9c8efb7ceb4c84dcbc776f08c3fc1"
server_key_version: "encryptionkey@0"
I20260809 05:21:31.512156 28427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:21:31.516016 28449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:31.517540 28427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 05:21:31.517714 28427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "c1db7285f5c144648514dbdb877b358a"
format_stamp: "Formatted at 2026-08-09 05:21:31 on dist-test-slave-rmhg"
server_key: "b7bffda819c01f358c1e8f7cc571e5a8"
server_key_iv: "9ea9c8efb7ceb4c84dcbc776f08c3fc1"
server_key_version: "encryptionkey@0"
I20260809 05:21:31.517915 28427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:31.540812 28427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:31.545392 28427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:31.545670 28427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:31.546717 28427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:31.548144 28427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:31.548218 28427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:31.548317 28427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:31.548370 28427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:31.579253 28427 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:39011
I20260809 05:21:31.579336 28633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:39011 every 8 connection(s)
I20260809 05:21:31.580617 28427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:31.586843 28226 heartbeater.cc:503] Master 127.22.1.126:38405 was elected leader, sending a full tablet report...
I20260809 05:21:31.587105 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 28427
I20260809 05:21:31.587277 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:31.587754 22533 external_mini_cluster.cc:1428] Setting key 9d95d78233ea351fa634a556ef5bcf82
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:31.601014 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.583254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:42129 (local address 127.22.1.126:38405)
0809 05:21:31.583462 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.583468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.584794 (+ 1326us) server_negotiation.cc:407] Connection header received
0809 05:21:31.586749 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.586755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.586844 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.586932 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.588856 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.589974 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.591953 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.592363 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.592462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.595855 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.595860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.595863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.595897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.598342 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.599043 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.599047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.599050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.599187 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.599528 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.599531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.599533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.599801 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.599921 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.600573 (+ 652us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.600747 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:21:31.602131 28636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.583832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38405 (local address 127.22.1.67:42129)
0809 05:21:31.584631 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:31.584681 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:31.586457 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:31.587214 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:31.587225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:31.587936 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:31.588651 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:31.588677 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.590127 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.590133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:31.591817 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:31.591831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.592566 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.592571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:31.592774 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.593456 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:31.593479 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:31.595640 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:31.598483 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:31.598490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:31.598505 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:31.598943 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:31.599270 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:31.599272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:31.599275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:31.599450 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:31.599900 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:31.599906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:31.600223 (+ 317us) client_negotiation.cc:769] Sending connection context
0809 05:21:31.600463 (+ 240us) client_negotiation.cc:240] Negotiation successful
0809 05:21:31.600887 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":306,"threads_started":1}
I20260809 05:21:31.603623 28634 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38405
I20260809 05:21:31.603888 28634 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:31.604595 28634 heartbeater.cc:511] Master 127.22.1.126:38405 requested a full tablet report, sending...
I20260809 05:21:31.605995 27930 ts_manager.cc:194] Registered new tserver with Master: c1db7285f5c144648514dbdb877b358a (127.22.1.67:39011)
I20260809 05:21:31.606796 27930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:42129
I20260809 05:21:31.615844 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:31.666960 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.652178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.22.1.65:36451)
0809 05:21:31.652793 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.652800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.652828 (+ 28us) server_negotiation.cc:407] Connection header received
0809 05:21:31.652890 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.652896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.653130 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.653344 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.654795 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.655858 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.657042 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.657691 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.657790 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.661721 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.661734 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.661753 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.661794 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.664420 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.665038 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.665046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.665048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.665203 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.665626 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.665630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.665632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.665996 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.666209 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.666461 (+ 252us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.666667 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:31.684579 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.671312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.22.1.126:38405)
0809 05:21:31.671530 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.671534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.671687 (+ 153us) server_negotiation.cc:407] Connection header received
0809 05:21:31.671833 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.671837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.671900 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.671973 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.673279 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.673988 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.675183 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.675591 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.675704 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.679481 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.679487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.679490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.679526 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.682020 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.682672 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.682677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.682680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.682812 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.683161 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.683165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.683167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.683489 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.683729 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.684033 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.684243 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260809 05:21:31.685247 27930 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51166
I20260809 05:21:31.686328 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 28026
I20260809 05:21:31.699292 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.687648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.22.1.65:36451)
0809 05:21:31.687812 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.687817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.688043 (+ 226us) server_negotiation.cc:407] Connection header received
0809 05:21:31.688177 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.688180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.688246 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.688322 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.689546 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.690236 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.691463 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.691908 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.692007 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.694328 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.694333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.694337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.694372 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.697095 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.697778 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.697781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.697784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.697904 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.698192 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.698194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.698196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.698437 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.698611 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.698879 (+ 268us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.699060 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 05:21:31.699905 28220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:31.782188 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 28026
I20260809 05:21:31.809458 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 28232
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:31.825769 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.810471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39076 (local address 127.22.1.66:39879)
0809 05:21:31.811008 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.811016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.811043 (+ 27us) server_negotiation.cc:407] Connection header received
0809 05:21:31.811103 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.811109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.811323 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.811473 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.812708 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.813815 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.815153 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.815765 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.815895 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.820233 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.820247 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.820270 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.820309 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.823184 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.823855 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.823862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.823865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.823994 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.824300 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.824303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.824305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.824689 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.824926 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.825202 (+ 276us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.825418 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":239,"threads_started":1}
I20260809 05:21:31.827085 28413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:31.908609 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 28232
I20260809 05:21:31.934463 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 28427
Aug 09 05:21:31 dist-test-slave-rmhg krb5kdc[27847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:31.950551 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:31.935509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.22.1.67:39011)
0809 05:21:31.936267 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:31.936274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:31.936303 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:21:31.936363 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:31.936369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:31.936590 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:31.936713 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:31.938164 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.939222 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.940468 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:31.941044 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:31.941141 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:31.945278 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:31.945288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:31.945305 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:31.945344 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:31.947868 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.948496 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.948504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.948506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.948634 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:31.948971 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:31.948975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:31.948977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:31.949369 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:31.949668 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:31.949951 (+ 283us) server_negotiation.cc:299] Negotiation successful
0809 05:21:31.950216 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":422,"threads_started":1}
I20260809 05:21:31.951797 28628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:32.033629 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 28427
I20260809 05:21:32.052955 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 27874
I20260809 05:21:32.054277 27977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:32.155653 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 27874
2026-08-09T05:21:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4086 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:32 dist-test-slave-rmhg krb5kdc[28658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:32 dist-test-slave-rmhg krb5kdc[28658](info): set up 2 sockets
Aug 09 05:21:32 dist-test-slave-rmhg krb5kdc[28658](info): commencing operation
krb5kdc: starting...
W20260809 05:21:33.346424 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:33 dist-test-slave-rmhg krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:33.568810 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39609
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:39609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:33.912608 28678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:33.912930 28678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:33.913023 28678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:33.920351 28678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:33.920485 28678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:33.920532 28678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:33.920595 28678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:33.920643 28678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:33.930696 28678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:39609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:33.932045 28678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:33.933368 28678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:33.945050 28691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:33.945091 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:33.945014 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:33.945815 28678 server_base.cc:1034] running on GCE node
I20260809 05:21:33.946627 28678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:33.948161 28678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:33.949390 28678 hybrid_clock.cc:648] HybridClock initialized: now 1786252893949327 us; error 44 us; skew 500 ppm
I20260809 05:21:33.949781 28678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:33 dist-test-slave-rmhg krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:33.952883 28678 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:33.954089 28678 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:33.954201 28678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:33.954533 28678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:33.957671 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "0cb34803c2944e138381aed710442de6"
format_stamp: "Formatted at 2026-08-09 05:21:33 on dist-test-slave-rmhg"
server_key: "824cac2886768cae1a6c3925a0f7da70"
server_key_iv: "29c34ee809fe4368a198112840e0720c"
server_key_version: "encryptionkey@0"
I20260809 05:21:33.958492 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "0cb34803c2944e138381aed710442de6"
format_stamp: "Formatted at 2026-08-09 05:21:33 on dist-test-slave-rmhg"
server_key: "824cac2886768cae1a6c3925a0f7da70"
server_key_iv: "29c34ee809fe4368a198112840e0720c"
server_key_version: "encryptionkey@0"
I20260809 05:21:33.963749 28678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:33.967442 28700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:33.968885 28678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:21:33.969095 28678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "0cb34803c2944e138381aed710442de6"
format_stamp: "Formatted at 2026-08-09 05:21:33 on dist-test-slave-rmhg"
server_key: "824cac2886768cae1a6c3925a0f7da70"
server_key_iv: "29c34ee809fe4368a198112840e0720c"
server_key_version: "encryptionkey@0"
I20260809 05:21:33.969357 28678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:33.993436 28678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:33.997799 28678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:33.998054 28678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:34.015277 28678 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:39609
I20260809 05:21:34.015301 28776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:39609 every 8 connection(s)
I20260809 05:21:34.016203 28678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:34.019919 28777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:34.021332 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 28678
I20260809 05:21:34.021545 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:34.022379 22533 external_mini_cluster.cc:1428] Setting key a8668602ac5ca6843046130f8addf05a
I20260809 05:21:34.034170 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: Bootstrap starting.
Aug 09 05:21:34 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:34.038159 28777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:34.039299 28777 log.cc:824] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:34.042596 28777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: No bootstrap required, opened a new log
I20260809 05:21:34.045980 28783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:34.024526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.22.1.126:39609)
0809 05:21:34.025300 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:34.025312 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:34.025363 (+ 51us) server_negotiation.cc:407] Connection header received
0809 05:21:34.026905 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:34.026928 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:34.027310 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:34.027634 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:34.029510 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.030828 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.032628 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.033373 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.033530 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:34.038280 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:34.038292 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:34.038313 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:34.038359 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:34.041641 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:34.042501 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:34.042508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:34.042514 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:34.042685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:34.043113 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:34.043117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:34.043119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:34.043691 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:34.043900 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:34.044396 (+ 496us) server_negotiation.cc:299] Negotiation successful
0809 05:21:34.044811 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":340,"threads_started":1}
I20260809 05:21:34.050729 28777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } }
I20260809 05:21:34.051074 28777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:34.051167 28777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb34803c2944e138381aed710442de6, State: Initialized, Role: FOLLOWER
I20260809 05:21:34.051687 28777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } }
I20260809 05:21:34.051898 28777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:34.052030 28777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:34.052177 28777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:34.054410 28777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } }
I20260809 05:21:34.054886 28777 leader_election.cc:302] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cb34803c2944e138381aed710442de6; no voters:
I20260809 05:21:34.055482 28777 leader_election.cc:288] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:34.055646 28785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:34.056537 28785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb34803c2944e138381aed710442de6, State: Running, Role: LEADER
I20260809 05:21:34.057071 28785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } }
I20260809 05:21:34.057826 28777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:34.060209 28788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cb34803c2944e138381aed710442de6. Latest consensus state: current_term: 1 leader_uuid: "0cb34803c2944e138381aed710442de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } } }
I20260809 05:21:34.060180 28787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cb34803c2944e138381aed710442de6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cb34803c2944e138381aed710442de6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39609 } } }
I20260809 05:21:34.060609 28788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:34.060613 28787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:34.062330 28797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:34.067564 28797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:34.077294 28797 catalog_manager.cc:1347] Generated new cluster ID: c35d21dffc3342e48032d5fef72804ad
I20260809 05:21:34.077436 28797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:34.103787 28797 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:34.105433 28797 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:34.120380 28797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: Generated new TSK 0
I20260809 05:21:34.121348 28797 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:34.181012 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:34.520200 28817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:34.520533 28817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:34.520681 28817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:34.528101 28817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:34.528245 28817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:34.528410 28817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:34.538599 28817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:34.540233 28817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:34.541850 28817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:34.555024 28822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:34.555068 28826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:34.554957 28823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:34.556125 28817 server_base.cc:1034] running on GCE node
I20260809 05:21:34.556798 28817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:34.557979 28817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:34.559229 28817 hybrid_clock.cc:648] HybridClock initialized: now 1786252894559182 us; error 38 us; skew 500 ppm
I20260809 05:21:34.559589 28817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:34 dist-test-slave-rmhg krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252894, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:34.562897 28817 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:34.564097 28817 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:34.564193 28817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:34.564558 28817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:34.567649 28817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "9ed4f6627dcf485d878a97a856e40fb7"
format_stamp: "Formatted at 2026-08-09 05:21:34 on dist-test-slave-rmhg"
server_key: "19235b5707a8ea204fa493615a30b1c0"
server_key_iv: "e94726e8a0fe39aebe77848864a5a8a1"
server_key_version: "encryptionkey@0"
I20260809 05:21:34.568441 28817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "9ed4f6627dcf485d878a97a856e40fb7"
format_stamp: "Formatted at 2026-08-09 05:21:34 on dist-test-slave-rmhg"
server_key: "19235b5707a8ea204fa493615a30b1c0"
server_key_iv: "e94726e8a0fe39aebe77848864a5a8a1"
server_key_version: "encryptionkey@0"
I20260809 05:21:34.574165 28817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:34.577880 28832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:34.579407 28817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:34.579612 28817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "9ed4f6627dcf485d878a97a856e40fb7"
format_stamp: "Formatted at 2026-08-09 05:21:34 on dist-test-slave-rmhg"
server_key: "19235b5707a8ea204fa493615a30b1c0"
server_key_iv: "e94726e8a0fe39aebe77848864a5a8a1"
server_key_version: "encryptionkey@0"
I20260809 05:21:34.579838 28817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:34.595742 28817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:34.599941 28817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:34.600219 28817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:34.601337 28817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:34.602772 28817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:34.602874 28817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:34.603003 28817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:34.603080 28817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:34.634084 28817 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:40875
I20260809 05:21:34.634130 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:40875 every 8 connection(s)
I20260809 05:21:34.634960 28817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:34.645294 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 28817
I20260809 05:21:34.645494 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:34.646157 22533 external_mini_cluster.cc:1428] Setting key 3309717d2d82c00a658eb94b701a9bea
Aug 09 05:21:34 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252894, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:34.656162 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:34.637365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:46309 (local address 127.22.1.126:39609)
0809 05:21:34.637822 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:34.637827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:34.638737 (+ 910us) server_negotiation.cc:407] Connection header received
0809 05:21:34.640584 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:34.640590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:34.640674 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:34.640781 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:34.642459 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.643189 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.644820 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.645271 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.645391 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:34.649748 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:34.649754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:34.649759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:34.649800 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:34.652826 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:34.653741 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:34.653747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:34.653751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:34.653901 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:34.654320 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:34.654324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:34.654327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:34.654683 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:34.654824 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:34.655589 (+ 765us) server_negotiation.cc:299] Negotiation successful
0809 05:21:34.655812 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260809 05:21:34.657234 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:34.637846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:39609 (local address 127.22.1.65:46309)
0809 05:21:34.638572 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:34.638612 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:34.640283 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:34.641056 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:34.641065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:34.641663 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:34.642265 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:34.642283 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.643350 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.643354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:34.644644 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:34.644658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:34.645515 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:34.645523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:34.645760 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:34.646484 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:34.646530 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:34.649482 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:34.652984 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:34.652992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:34.653009 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:34.653598 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:34.654015 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:34.654018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:34.654021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:34.654214 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:34.654822 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:34.654829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:34.655191 (+ 362us) client_negotiation.cc:769] Sending connection context
0809 05:21:34.655443 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:21:34.655883 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":273,"threads_started":1}
I20260809 05:21:34.658883 28996 heartbeater.cc:347] Connected to a master server at 127.22.1.126:39609
I20260809 05:21:34.659217 28996 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:34.659905 28996 heartbeater.cc:511] Master 127.22.1.126:39609 requested a full tablet report, sending...
I20260809 05:21:34.662035 28728 ts_manager.cc:194] Registered new tserver with Master: 9ed4f6627dcf485d878a97a856e40fb7 (127.22.1.65:40875)
I20260809 05:21:34.663508 28728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:46309
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:34.694594 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:35.026856 29004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:35.027143 29004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:35.027278 29004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:35.034508 29004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:35.034639 29004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:35.034797 29004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:35.044930 29004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:35.046370 29004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:35.047717 29004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:35.060583 29012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:35.060619 29016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:35.060640 29009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:35.061245 29004 server_base.cc:1034] running on GCE node
I20260809 05:21:35.061931 29004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:35.062844 29004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:35.064044 29004 hybrid_clock.cc:648] HybridClock initialized: now 1786252895063998 us; error 35 us; skew 500 ppm
I20260809 05:21:35.064369 29004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:35 dist-test-slave-rmhg krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252895, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:35.070766 29004 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:35.072065 29004 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:35.072167 29004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:35.072497 29004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:35.075711 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "be7df03fd17b45dcb2f36f93a24edcad"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "23ca109cfb9156e7e53fb91e3baaa99a"
server_key_iv: "ae825c00313ccc346110c8de63e3744c"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.076493 29004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "be7df03fd17b45dcb2f36f93a24edcad"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "23ca109cfb9156e7e53fb91e3baaa99a"
server_key_iv: "ae825c00313ccc346110c8de63e3744c"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.081892 29004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260809 05:21:35.085502 29032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:35.086825 29004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:35.086973 29004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "be7df03fd17b45dcb2f36f93a24edcad"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "23ca109cfb9156e7e53fb91e3baaa99a"
server_key_iv: "ae825c00313ccc346110c8de63e3744c"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.087172 29004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:35.114092 29004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:35.118438 29004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:35.118700 29004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:35.119680 29004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:35.121090 29004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:35.121206 29004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 05:21:35.121310 29004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:35.121379 29004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:35.151909 29004 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:42649
I20260809 05:21:35.151963 29186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:42649 every 8 connection(s)
I20260809 05:21:35.152760 29004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:35.156199 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29004
I20260809 05:21:35.156374 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:35.156935 22533 external_mini_cluster.cc:1428] Setting key 09e03ab6d1bb7ccdcf159334118083b0
Aug 09 05:21:35 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252895, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:35.173058 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.155227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:52069 (local address 127.22.1.126:39609)
0809 05:21:35.155401 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.155406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.156681 (+ 1275us) server_negotiation.cc:407] Connection header received
0809 05:21:35.158735 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.158741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.158826 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.158911 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:35.160567 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.161329 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.162934 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.163394 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.163507 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.167102 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:35.167108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:35.167112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:35.167147 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:35.169763 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.170643 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.170647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.170649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.170771 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.171257 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.171260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.171262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.171528 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:35.171655 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.172599 (+ 944us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.172812 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:21:35.174574 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.155661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:39609 (local address 127.22.1.66:52069)
0809 05:21:35.156486 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.156553 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.158453 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.159242 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.159251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.159782 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:35.160376 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.160405 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.161475 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.161479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.162786 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.162799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.163629 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.163637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.163856 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.164520 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:35.164545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:35.166856 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:35.169920 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:35.169930 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:35.169950 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:35.170541 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:35.170893 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:35.170898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:35.170902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:35.171168 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:35.171663 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:35.171671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:35.172116 (+ 445us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.172480 (+ 364us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.173067 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":337,"threads_started":1}
I20260809 05:21:35.176121 29188 heartbeater.cc:347] Connected to a master server at 127.22.1.126:39609
I20260809 05:21:35.176468 29188 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:35.177330 29188 heartbeater.cc:511] Master 127.22.1.126:39609 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:21:35.178927 28728 ts_manager.cc:194] Registered new tserver with Master: be7df03fd17b45dcb2f36f93a24edcad (127.22.1.66:42649)
I20260809 05:21:35.179744 28728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:52069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:35.206182 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:35.533449 29194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:35.533784 29194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:35.533914 29194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:35.541203 29194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:35.541352 29194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:35.541517 29194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:35.551644 29194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:39609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:35.552996 29194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:35.554322 29194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:35.566958 29202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:35.566984 29203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:35.567266 29205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:35.567942 29194 server_base.cc:1034] running on GCE node
I20260809 05:21:35.568526 29194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:35.569517 29194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:35.570720 29194 hybrid_clock.cc:648] HybridClock initialized: now 1786252895570666 us; error 52 us; skew 500 ppm
I20260809 05:21:35.571025 29194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:35 dist-test-slave-rmhg krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252895, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:35.574241 29194 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:35.575371 29194 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:35.575475 29194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:35.575788 29194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:35.578689 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "97b014aab3d24e1aa232f0d9fbb87d50"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "54867cb595f8b48c233873b6cc89a785"
server_key_iv: "da4b56e84db963646517a3e6ae8ba821"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.579449 29194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "97b014aab3d24e1aa232f0d9fbb87d50"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "54867cb595f8b48c233873b6cc89a785"
server_key_iv: "da4b56e84db963646517a3e6ae8ba821"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.584283 29194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:21:35.587695 29213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:35.588920 29194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:21:35.589077 29194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "97b014aab3d24e1aa232f0d9fbb87d50"
format_stamp: "Formatted at 2026-08-09 05:21:35 on dist-test-slave-rmhg"
server_key: "54867cb595f8b48c233873b6cc89a785"
server_key_iv: "da4b56e84db963646517a3e6ae8ba821"
server_key_version: "encryptionkey@0"
I20260809 05:21:35.589361 29194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:35.612701 29194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:35.616532 29194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:35.616798 29194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:35.617790 29194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:35.619146 29194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:35.619248 29194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:35.619354 29194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:35.619426 29194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:35.648998 29194 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:42433
I20260809 05:21:35.649050 29367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:42433 every 8 connection(s)
I20260809 05:21:35.649848 29194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:35.658222 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29194
I20260809 05:21:35.658433 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:35.658865 22533 external_mini_cluster.cc:1428] Setting key 7eac569fbfd29ea60912599ce6a38daf
Aug 09 05:21:35 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252895, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:35.666633 28996 heartbeater.cc:503] Master 127.22.1.126:39609 was elected leader, sending a full tablet report...
I20260809 05:21:35.669054 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.652261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:53613 (local address 127.22.1.126:39609)
0809 05:21:35.652461 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.652467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.653552 (+ 1085us) server_negotiation.cc:407] Connection header received
0809 05:21:35.655291 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.655296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.655393 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.655485 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:35.656922 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.657708 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.659480 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.659982 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.660108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.663669 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:35.663674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:35.663677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:35.663710 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:35.666044 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.666753 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.666759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.666762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.666974 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.667402 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.667407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.667410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.667781 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:35.667936 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.668641 (+ 705us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.668816 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:21:35.670142 29370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.652690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:39609 (local address 127.22.1.67:53613)
0809 05:21:35.653396 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.653434 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.655038 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.655692 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.655699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.656199 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:35.656757 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.656774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.657833 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.657837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.659333 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.659346 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.660325 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.660331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.660538 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.661245 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:35.661268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:35.663478 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:35.666159 (+ 2681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:35.666167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:35.666182 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:35.666631 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:35.667089 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:35.667094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:35.667097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:35.667286 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:35.667915 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:35.667921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:35.668301 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.668551 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.668956 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":263,"threads_started":1}
I20260809 05:21:35.671463 29368 heartbeater.cc:347] Connected to a master server at 127.22.1.126:39609
I20260809 05:21:35.671738 29368 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:35.672338 29368 heartbeater.cc:511] Master 127.22.1.126:39609 requested a full tablet report, sending...
I20260809 05:21:35.673758 28728 ts_manager.cc:194] Registered new tserver with Master: 97b014aab3d24e1aa232f0d9fbb87d50 (127.22.1.67:42433)
I20260809 05:21:35.674417 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:35.674655 28728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:53613
I20260809 05:21:35.690454 29000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.679383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.22.1.126:39609)
0809 05:21:35.679566 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.679573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.679984 (+ 411us) server_negotiation.cc:407] Connection header received
0809 05:21:35.680116 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.680120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.680185 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.680259 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:35.681618 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.682352 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.683540 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.683921 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.684007 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.686131 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:35.686143 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:35.686146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:35.686174 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:35.688306 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.688847 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.688850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.688852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.688960 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.689290 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.689293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.689295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.689596 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:35.689768 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.690019 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.690173 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:21:35.694180 28728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:35.700666 28728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:35.723160 29390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.712551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:42433 (local address 127.0.0.1:55828)
0809 05:21:35.713680 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.713697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.713800 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.716066 (+ 2266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.716071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.716100 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:35.716518 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.716530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.718705 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.718711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.720400 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.720416 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.722102 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.722289 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.722337 (+ 48us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.722476 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.722680 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":54,"thread_start_us":3338,"threads_started":1}
I20260809 05:21:35.723290 29386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.711951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40875 (local address 127.0.0.1:33280)
0809 05:21:35.712825 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.712860 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.713049 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.715534 (+ 2485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.715541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.715577 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:35.715965 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.715979 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.717956 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.717962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.720562 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.720577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.722485 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.722669 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.722698 (+ 29us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.722779 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.722910 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":120,"thread_start_us":512,"threads_started":1}
I20260809 05:21:35.723789 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.713240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.22.1.67:42433)
0809 05:21:35.715444 (+ 2204us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.715453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.715488 (+ 35us) server_negotiation.cc:407] Connection header received
0809 05:21:35.715603 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.715611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.715846 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.716735 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:35.717021 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.718562 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.720644 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.721905 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722025 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.722267 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.723106 (+ 839us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.723445 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2029,"thread_start_us":1847,"threads_started":1}
I20260809 05:21:35.723999 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.712287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.22.1.65:40875)
0809 05:21:35.715047 (+ 2760us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.715053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.715076 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:21:35.715152 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.715158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.715340 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.715503 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:35.716209 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.717800 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.720755 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.722233 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722414 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.722642 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.723614 (+ 972us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.723740 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2619,"thread_start_us":2487,"threads_started":1}
I20260809 05:21:35.725359 29388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.713240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42649 (local address 127.0.0.1:41406)
0809 05:21:35.716185 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.716201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.716311 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.716775 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.716779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.716802 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:35.717212 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.717229 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.718831 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.718837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.721063 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.721077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.722885 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.723076 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.723110 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.723203 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.724071 (+ 868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"mutex_wait_us":1,"thread_start_us":697,"threads_started":1}
I20260809 05:21:35.725412 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.713756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.22.1.66:42649)
0809 05:21:35.714295 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.714303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.716291 (+ 1988us) server_negotiation.cc:407] Connection header received
0809 05:21:35.716447 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.716453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.716620 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.716770 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:35.717565 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.718683 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.721280 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.722666 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.722788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.722975 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.723849 (+ 874us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.725085 (+ 1236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":701,"threads_started":1}
I20260809 05:21:35.727861 28907 tablet_service.cc:1467] Processing CreateTablet for tablet 60f2403d986d4d10bde37c20c76e110f (DEFAULT_TABLE table=test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:35.727957 29277 tablet_service.cc:1467] Processing CreateTablet for tablet 60f2403d986d4d10bde37c20c76e110f (DEFAULT_TABLE table=test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:35.729136 28907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f2403d986d4d10bde37c20c76e110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:35.728684 29097 tablet_service.cc:1467] Processing CreateTablet for tablet 60f2403d986d4d10bde37c20c76e110f (DEFAULT_TABLE table=test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:35.729336 29277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f2403d986d4d10bde37c20c76e110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:35.729988 29097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60f2403d986d4d10bde37c20c76e110f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:35.740463 29393 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Bootstrap starting.
I20260809 05:21:35.743602 29398 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Bootstrap starting.
I20260809 05:21:35.745779 29393 tablet_bootstrap.cc:654] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:35.745857 29396 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Bootstrap starting.
I20260809 05:21:35.747272 29398 tablet_bootstrap.cc:654] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:35.747295 29393 log.cc:824] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:35.748416 29398 log.cc:824] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:35.748844 29396 tablet_bootstrap.cc:654] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:35.749998 29396 log.cc:824] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:35.750725 29393 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: No bootstrap required, opened a new log
I20260809 05:21:35.751042 29393 ts_tablet_manager.cc:1397] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 05:21:35.751880 29398 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: No bootstrap required, opened a new log
I20260809 05:21:35.752156 29398 ts_tablet_manager.cc:1397] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260809 05:21:35.752663 29396 tablet_bootstrap.cc:492] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: No bootstrap required, opened a new log
I20260809 05:21:35.753048 29396 ts_tablet_manager.cc:1397] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260809 05:21:35.760280 29396 raft_consensus.cc:348] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.760669 29396 raft_consensus.cc:374] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:35.760774 29396 raft_consensus.cc:729] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b014aab3d24e1aa232f0d9fbb87d50, State: Initialized, Role: FOLLOWER
I20260809 05:21:35.761528 29396 consensus_queue.cc:260] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.762727 29368 heartbeater.cc:503] Master 127.22.1.126:39609 was elected leader, sending a full tablet report...
I20260809 05:21:35.762534 29393 raft_consensus.cc:348] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.762984 29393 raft_consensus.cc:374] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:35.763078 29393 raft_consensus.cc:729] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed4f6627dcf485d878a97a856e40fb7, State: Initialized, Role: FOLLOWER
I20260809 05:21:35.763473 29396 ts_tablet_manager.cc:1428] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260809 05:21:35.763177 29398 raft_consensus.cc:348] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.763928 29398 raft_consensus.cc:374] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:35.763785 29393 consensus_queue.cc:260] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.764045 29398 raft_consensus.cc:729] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be7df03fd17b45dcb2f36f93a24edcad, State: Initialized, Role: FOLLOWER
I20260809 05:21:35.764655 29398 consensus_queue.cc:260] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.768553 29398 ts_tablet_manager.cc:1428] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260809 05:21:35.769021 29188 heartbeater.cc:503] Master 127.22.1.126:39609 was elected leader, sending a full tablet report...
I20260809 05:21:35.771750 29393 ts_tablet_manager.cc:1428] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260809 05:21:35.850524 29405 raft_consensus.cc:482] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:35.850837 29405 raft_consensus.cc:504] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.852627 29405 leader_election.cc:288] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be7df03fd17b45dcb2f36f93a24edcad (127.22.1.66:42649), 97b014aab3d24e1aa232f0d9fbb87d50 (127.22.1.67:42433)
I20260809 05:21:35.859570 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.853192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:43983 (local address 127.22.1.67:42433)
0809 05:21:35.853468 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.853475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.854177 (+ 702us) server_negotiation.cc:407] Connection header received
0809 05:21:35.854372 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.854376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.854436 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.854513 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:35.855161 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.855869 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.857744 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.858536 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.858657 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.858761 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.859203 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.859327 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260809 05:21:35.859573 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.853018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:46597 (local address 127.22.1.66:42649)
0809 05:21:35.853254 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.853260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.854090 (+ 830us) server_negotiation.cc:407] Connection header received
0809 05:21:35.854269 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.854274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.854334 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.854407 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:35.855037 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.855747 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.857730 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.858525 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.858651 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.858756 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.859191 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.859321 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:21:35.859714 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.853142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:42433 (local address 127.22.1.65:43983)
0809 05:21:35.854077 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.854097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.854235 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.854553 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.854557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.854599 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:35.854990 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.855000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.856013 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.856019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.857584 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.857597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.858880 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.858888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.859051 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.859075 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.859157 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.859282 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":64,"thread_start_us":511,"threads_started":1}
I20260809 05:21:35.860145 29409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.853096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:42649 (local address 127.22.1.65:46597)
0809 05:21:35.853953 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:35.853973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:35.854114 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:35.854450 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:35.854454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:35.854480 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:35.854876 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.854888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.855876 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.855880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.857577 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:35.857591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.858767 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.858775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:35.859034 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.859060 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 05:21:35.859144 (+ 84us) client_negotiation.cc:240] Negotiation successful
0809 05:21:35.859271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":53,"thread_start_us":485,"threads_started":1}
I20260809 05:21:35.860777 29303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f2403d986d4d10bde37c20c76e110f" candidate_uuid: "9ed4f6627dcf485d878a97a856e40fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" is_pre_election: true
I20260809 05:21:35.861081 29123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f2403d986d4d10bde37c20c76e110f" candidate_uuid: "9ed4f6627dcf485d878a97a856e40fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7df03fd17b45dcb2f36f93a24edcad" is_pre_election: true
I20260809 05:21:35.861331 29303 raft_consensus.cc:2454] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed4f6627dcf485d878a97a856e40fb7 in term 0.
I20260809 05:21:35.861601 29123 raft_consensus.cc:2454] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ed4f6627dcf485d878a97a856e40fb7 in term 0.
I20260809 05:21:35.862115 28834 leader_election.cc:302] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97b014aab3d24e1aa232f0d9fbb87d50, 9ed4f6627dcf485d878a97a856e40fb7; no voters:
I20260809 05:21:35.862519 29405 raft_consensus.cc:2781] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:35.862674 29405 raft_consensus.cc:482] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:35.862779 29405 raft_consensus.cc:3037] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:35.865011 29405 raft_consensus.cc:504] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.865645 29405 leader_election.cc:288] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [CANDIDATE]: Term 1 election: Requested vote from peers be7df03fd17b45dcb2f36f93a24edcad (127.22.1.66:42649), 97b014aab3d24e1aa232f0d9fbb87d50 (127.22.1.67:42433)
I20260809 05:21:35.866111 29123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f2403d986d4d10bde37c20c76e110f" candidate_uuid: "9ed4f6627dcf485d878a97a856e40fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be7df03fd17b45dcb2f36f93a24edcad"
I20260809 05:21:35.866196 29303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60f2403d986d4d10bde37c20c76e110f" candidate_uuid: "9ed4f6627dcf485d878a97a856e40fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97b014aab3d24e1aa232f0d9fbb87d50"
I20260809 05:21:35.866319 29123 raft_consensus.cc:3037] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:35.866405 29303 raft_consensus.cc:3037] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:35.868985 29123 raft_consensus.cc:2454] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed4f6627dcf485d878a97a856e40fb7 in term 1.
I20260809 05:21:35.869007 29303 raft_consensus.cc:2454] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ed4f6627dcf485d878a97a856e40fb7 in term 1.
I20260809 05:21:35.869396 28836 leader_election.cc:302] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ed4f6627dcf485d878a97a856e40fb7, be7df03fd17b45dcb2f36f93a24edcad; no voters:
I20260809 05:21:35.869710 29405 raft_consensus.cc:2781] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:35.870163 29405 raft_consensus.cc:686] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed4f6627dcf485d878a97a856e40fb7, State: Running, Role: LEADER
I20260809 05:21:35.870700 29405 consensus_queue.cc:237] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } }
I20260809 05:21:35.875262 28728 catalog_manager.cc:5495] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ed4f6627dcf485d878a97a856e40fb7 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "9ed4f6627dcf485d878a97a856e40fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ed4f6627dcf485d878a97a856e40fb7" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 } health_report { overall_health: UNKNOWN } } }
W20260809 05:21:35.887039 28999 tablet.cc:2369] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:35.901608 29369 tablet.cc:2369] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:35.904624 29190 tablet.cc:2369] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:35.917820 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.912249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.22.1.65:40875)
0809 05:21:35.912483 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.912489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.912551 (+ 62us) server_negotiation.cc:407] Connection header received
0809 05:21:35.912731 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.912735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.912825 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.912948 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:35.913334 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.914087 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.915443 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.916039 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.916165 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.916669 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:35.917234 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:35.917376 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.917470 (+ 94us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.917558 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:21:35.926266 29303 raft_consensus.cc:1261] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 1 FOLLOWER]: Refusing update from remote peer 9ed4f6627dcf485d878a97a856e40fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:35.926266 29123 raft_consensus.cc:1261] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 1 FOLLOWER]: Refusing update from remote peer 9ed4f6627dcf485d878a97a856e40fb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:35.927407 29405 consensus_queue.cc:1035] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be7df03fd17b45dcb2f36f93a24edcad" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:35.927763 29411 consensus_queue.cc:1035] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97b014aab3d24e1aa232f0d9fbb87d50" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 42433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:35.938374 29415 mvcc.cc:205] Tried to move back new op lower bound from 7316491861703520256 to 7316491861490098176. Current Snapshot: MvccSnapshot[applied={T|T < 7316491861703520256}]
I20260809 05:21:35.939805 29416 mvcc.cc:205] Tried to move back new op lower bound from 7316491861703520256 to 7316491861490098176. Current Snapshot: MvccSnapshot[applied={T|T < 7316491861703520256}]
I20260809 05:21:35.942355 29417 mvcc.cc:205] Tried to move back new op lower bound from 7316491861703520256 to 7316491861490098176. Current Snapshot: MvccSnapshot[applied={T|T < 7316491861703520256}]
I20260809 05:21:35.950963 28727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:35.951225 28727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:35.955004 28727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cb34803c2944e138381aed710442de6: Sending DeleteTablet for 3 replicas of tablet 60f2403d986d4d10bde37c20c76e110f
I20260809 05:21:35.956322 28907 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f2403d986d4d10bde37c20c76e110f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:33280
I20260809 05:21:35.956321 29097 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f2403d986d4d10bde37c20c76e110f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41406
I20260809 05:21:35.956544 29277 tablet_service.cc:1514] Processing DeleteTablet for tablet 60f2403d986d4d10bde37c20c76e110f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55828
I20260809 05:21:35.957412 29427 tablet_replica.cc:330] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: stopping tablet replica
I20260809 05:21:35.957623 29425 tablet_replica.cc:330] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: stopping tablet replica
I20260809 05:21:35.957819 29427 raft_consensus.cc:2229] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:35.958063 29425 raft_consensus.cc:2229] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:35.958356 29427 raft_consensus.cc:2258] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:35.958557 29425 raft_consensus.cc:2258] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:35.958599 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 28817
I20260809 05:21:35.960988 29425 ts_tablet_manager.cc:1905] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:35.966719 29427 ts_tablet_manager.cc:1905] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:35.968101 29425 ts_tablet_manager.cc:1918] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:35.968279 29425 log.cc:1196] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/60f2403d986d4d10bde37c20c76e110f
I20260809 05:21:35.968850 29425 ts_tablet_manager.cc:1939] T 60f2403d986d4d10bde37c20c76e110f P be7df03fd17b45dcb2f36f93a24edcad: Deleting consensus metadata
I20260809 05:21:35.969758 29429 tablet_replica.cc:330] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: stopping tablet replica
I20260809 05:21:35.970217 29429 raft_consensus.cc:2229] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:35.970933 29429 raft_consensus.cc:2258] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:35.973202 28707 catalog_manager.cc:4841] TS be7df03fd17b45dcb2f36f93a24edcad (127.22.1.66:42649): tablet 60f2403d986d4d10bde37c20c76e110f (table test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]) successfully deleted
I20260809 05:21:35.973366 29429 ts_tablet_manager.cc:1905] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:35.977210 29427 ts_tablet_manager.cc:1918] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:35.977573 29427 log.cc:1196] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/60f2403d986d4d10bde37c20c76e110f
I20260809 05:21:35.978169 29427 ts_tablet_manager.cc:1939] T 60f2403d986d4d10bde37c20c76e110f P 97b014aab3d24e1aa232f0d9fbb87d50: Deleting consensus metadata
I20260809 05:21:35.980149 28706 catalog_manager.cc:4841] TS 97b014aab3d24e1aa232f0d9fbb87d50 (127.22.1.67:42433): tablet 60f2403d986d4d10bde37c20c76e110f (table test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]) successfully deleted
I20260809 05:21:35.980649 29429 ts_tablet_manager.cc:1918] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:35.980808 29429 log.cc:1196] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/60f2403d986d4d10bde37c20c76e110f
I20260809 05:21:35.981309 29429 ts_tablet_manager.cc:1939] T 60f2403d986d4d10bde37c20c76e110f P 9ed4f6627dcf485d878a97a856e40fb7: Deleting consensus metadata
I20260809 05:21:35.982750 28707 catalog_manager.cc:4841] TS 9ed4f6627dcf485d878a97a856e40fb7 (127.22.1.65:40875): tablet 60f2403d986d4d10bde37c20c76e110f (table test-table [id=5eb778877f0b4e8782c9b04e3d2588e8]) successfully deleted
Aug 09 05:21:35 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:35.991444 29387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:35.959919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.22.1.65:40875)
0809 05:21:35.960102 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:35.960108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:35.970882 (+ 10774us) server_negotiation.cc:407] Connection header received
0809 05:21:35.971070 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:35.971076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:35.971154 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:35.971239 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:35.974982 (+ 3743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.976249 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.978186 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:35.978942 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:35.979050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:35.984169 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:35.984184 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:35.984204 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:35.984253 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:35.988168 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.988920 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.988930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.988934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.989118 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:35.989626 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:35.989631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:35.989635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:35.990115 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:35.990430 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:35.990744 (+ 314us) server_negotiation.cc:299] Negotiation successful
0809 05:21:35.990990 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 05:21:35.992151 28989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:36.088470 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 28817
I20260809 05:21:36.112110 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29004
Aug 09 05:21:36 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:36.126433 29392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:36.112841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.22.1.66:42649)
0809 05:21:36.113003 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:36.113008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:36.113041 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:21:36.113222 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:36.113226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:36.113291 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:36.113373 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:36.114616 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:36.115375 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:36.116596 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:36.116982 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:36.117078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:36.121259 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:36.121274 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:36.121292 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:36.121332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:36.124096 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:36.124677 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:36.124684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:36.124686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:36.124812 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:36.125098 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:36.125101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:36.125103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:36.125520 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:36.125746 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:36.125942 (+ 196us) server_negotiation.cc:299] Negotiation successful
0809 05:21:36.126111 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:21:36.127162 29181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:36.215590 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29004
I20260809 05:21:36.239042 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29194
Aug 09 05:21:36 dist-test-slave-rmhg krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:36.253736 29391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:36.239805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.22.1.67:42433)
0809 05:21:36.239983 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:36.239989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:36.240033 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:21:36.240233 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:36.240237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:36.240302 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:36.240401 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:36.241764 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:36.242500 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:36.243789 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:36.244217 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:36.244333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:36.248613 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:36.248628 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:36.248647 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:36.248683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:36.251313 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:36.251955 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:36.251964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:36.251968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:36.252121 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:36.252437 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:36.252440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:36.252442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:36.252778 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:36.252998 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:36.253302 (+ 304us) server_negotiation.cc:299] Negotiation successful
0809 05:21:36.253477 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:21:36.254422 29362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:36.342978 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29194
I20260809 05:21:36.370224 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 28678
I20260809 05:21:36.371393 28770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:36.480574 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 28678
2026-08-09T05:21:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4325 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-09T05:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:36Z Disabled control of system clock
I20260809 05:21:36.556349 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:46529
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:46529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 05:21:36.886241 29440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:36.886523 29440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:36.886631 29440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:36.893647 29440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:36.893743 29440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:36.893800 29440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:36.893877 29440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:36.893931 29440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:36.902796 29440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:46529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:46529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:36.904057 29440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:36.905934 29440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:36.916924 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:36.916992 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:36.917438 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:36.917740 29440 server_base.cc:1034] running on GCE node
I20260809 05:21:36.918493 29440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:36.919839 29440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:36.921061 29440 hybrid_clock.cc:648] HybridClock initialized: now 1786252896921019 us; error 31 us; skew 500 ppm
I20260809 05:21:36.921432 29440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:36.923689 29440 webserver.cc:460] Webserver started at http://127.22.1.126:45075/ using document root <none> and password file <none>
I20260809 05:21:36.924396 29440 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:36.924517 29440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:36.924832 29440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:36.927825 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "15cd2325123f43e28f1639c5e22e1bad"
format_stamp: "Formatted at 2026-08-09 05:21:36 on dist-test-slave-rmhg"
server_key: "6b7a9839a999e3aefcf9e84f1d6ff311"
server_key_iv: "bae492c861df2a3f7908432fd6f4668a"
server_key_version: "encryptionkey@0"
I20260809 05:21:36.928622 29440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "15cd2325123f43e28f1639c5e22e1bad"
format_stamp: "Formatted at 2026-08-09 05:21:36 on dist-test-slave-rmhg"
server_key: "6b7a9839a999e3aefcf9e84f1d6ff311"
server_key_iv: "bae492c861df2a3f7908432fd6f4668a"
server_key_version: "encryptionkey@0"
I20260809 05:21:36.933705 29440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 05:21:36.937111 29459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:36.938463 29440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:36.938624 29440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "15cd2325123f43e28f1639c5e22e1bad"
format_stamp: "Formatted at 2026-08-09 05:21:36 on dist-test-slave-rmhg"
server_key: "6b7a9839a999e3aefcf9e84f1d6ff311"
server_key_iv: "bae492c861df2a3f7908432fd6f4668a"
server_key_version: "encryptionkey@0"
I20260809 05:21:36.938822 29440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:36.944447 29440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:36.945255 29440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:36.945533 29440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:36.962113 29440 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:46529
I20260809 05:21:36.962148 29532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:46529 every 8 connection(s)
I20260809 05:21:36.963445 29440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:36.966351 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29440
I20260809 05:21:36.966559 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:36.967216 22533 external_mini_cluster.cc:1428] Setting key 4150b21383b3c984d6d3c2653745d93b
I20260809 05:21:36.967569 29533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:36.976842 29539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:36.969268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.22.1.126:46529)
0809 05:21:36.970417 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:36.970432 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:36.970489 (+ 57us) server_negotiation.cc:407] Connection header received
0809 05:21:36.972576 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:36.972605 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:36.973232 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:36.973701 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:36.974748 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:36.974761 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:36.974783 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:36.974804 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:36.974880 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:36.974903 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:36.975152 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:36.975298 (+ 146us) server_negotiation.cc:299] Negotiation successful
0809 05:21:36.975587 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":330,"threads_started":1}
I20260809 05:21:36.982432 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: Bootstrap starting.
I20260809 05:21:36.985859 29533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:36.986866 29533 log.cc:824] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:36.990034 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: No bootstrap required, opened a new log
I20260809 05:21:36.997527 29533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } }
I20260809 05:21:36.997897 29533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:36.997970 29533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15cd2325123f43e28f1639c5e22e1bad, State: Initialized, Role: FOLLOWER
I20260809 05:21:36.998494 29533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } }
I20260809 05:21:36.998692 29533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:36.998788 29533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:36.998905 29533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:37.000988 29533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } }
I20260809 05:21:37.001514 29533 leader_election.cc:302] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15cd2325123f43e28f1639c5e22e1bad; no voters:
I20260809 05:21:37.002094 29533 leader_election.cc:288] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:37.002219 29543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:37.002933 29543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [term 1 LEADER]: Becoming Leader. State: Replica: 15cd2325123f43e28f1639c5e22e1bad, State: Running, Role: LEADER
I20260809 05:21:37.003504 29543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } }
I20260809 05:21:37.004233 29533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:37.006469 29548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15cd2325123f43e28f1639c5e22e1bad. Latest consensus state: current_term: 1 leader_uuid: "15cd2325123f43e28f1639c5e22e1bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } } }
I20260809 05:21:37.006539 29547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15cd2325123f43e28f1639c5e22e1bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15cd2325123f43e28f1639c5e22e1bad" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46529 } } }
I20260809 05:21:37.006879 29548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:37.006929 29547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:37.007898 29557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:37.013720 29557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:37.023304 29557 catalog_manager.cc:1347] Generated new cluster ID: d8e63d7e4b0e48e283817ab9907b7119
I20260809 05:21:37.023430 29557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:37.045938 29557 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:37.046470 29557 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:37.064780 29557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: Generated new TSK 0
I20260809 05:21:37.066035 29557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:21:37.082052 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--builtin_ntp_servers=127.22.1.84:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260809 05:21:37.422924 29572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:37.423233 29572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:37.423337 29572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:37.430980 29572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:37.431088 29572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:37.431206 29572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:37.440600 29572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:37.441931 29572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:37.443876 29572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:37.457419 29581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:37.457350 29579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:37.457460 29585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:37.458220 29572 server_base.cc:1034] running on GCE node
I20260809 05:21:37.458935 29572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:37.459936 29572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:37.461143 29572 hybrid_clock.cc:648] HybridClock initialized: now 1786252897461084 us; error 56 us; skew 500 ppm
I20260809 05:21:37.461488 29572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:37.463805 29572 webserver.cc:460] Webserver started at http://127.22.1.65:42959/ using document root <none> and password file <none>
I20260809 05:21:37.464552 29572 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:37.464668 29572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:37.464995 29572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:37.468024 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "6f785d6797194f42abf7770ec9a6b539"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "c6a1f1d6331c101555a8f3320dc59e75"
server_key_iv: "56a6068037a7c01844a27b71606aa83e"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.468890 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "6f785d6797194f42abf7770ec9a6b539"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "c6a1f1d6331c101555a8f3320dc59e75"
server_key_iv: "56a6068037a7c01844a27b71606aa83e"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.474519 29572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:21:37.478487 29595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.479933 29572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:37.480103 29572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "6f785d6797194f42abf7770ec9a6b539"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "c6a1f1d6331c101555a8f3320dc59e75"
server_key_iv: "56a6068037a7c01844a27b71606aa83e"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.480319 29572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:37.486333 29572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:37.487201 29572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:37.487466 29572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:37.488471 29572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:37.489980 29572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:37.490079 29572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.490188 29572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:37.490257 29572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.520541 29572 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:38935
I20260809 05:21:37.520627 29763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:38935 every 8 connection(s)
I20260809 05:21:37.522007 29572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:37.524791 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29572
I20260809 05:21:37.524998 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:37.525678 22533 external_mini_cluster.cc:1428] Setting key ec8bdbfc19363a3f7f82d91827efb45f
I20260809 05:21:37.528724 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--builtin_ntp_servers=127.22.1.84:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 05:21:37.530599 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:37.524631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:37661 (local address 127.22.1.126:46529)
0809 05:21:37.525145 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:37.525170 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:37.526500 (+ 1330us) server_negotiation.cc:407] Connection header received
0809 05:21:37.528389 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:37.528395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:37.528465 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:37.528589 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:37.529883 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:37.529889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:37.529893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:37.529913 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:37.529975 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:37.529992 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:37.530122 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:37.530250 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 05:21:37.530311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
I20260809 05:21:37.531512 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:37.525254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46529 (local address 127.22.1.65:37661)
0809 05:21:37.526330 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:37.526384 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:37.528117 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:37.528776 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:37.528784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:37.528893 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:37.529565 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:37.529568 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:37.529616 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:37.529709 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:37.530080 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:37.530087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:37.530093 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:21:37.530306 (+ 213us) client_negotiation.cc:240] Negotiation successful
0809 05:21:37.530523 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":325,"threads_started":1}
I20260809 05:21:37.533052 29764 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46529
I20260809 05:21:37.533430 29764 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:37.534183 29764 heartbeater.cc:511] Master 127.22.1.126:46529 requested a full tablet report, sending...
I20260809 05:21:37.536615 29484 ts_manager.cc:194] Registered new tserver with Master: 6f785d6797194f42abf7770ec9a6b539 (127.22.1.65:38935)
W20260809 05:21:37.860842 29769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:37.861111 29769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:37.861235 29769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:37.868330 29769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:37.868436 29769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:37.868551 29769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:37.877636 29769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:37.878901 29769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:37.880990 29769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:37.893672 29776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:37.893631 29779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:37.893704 29777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:37.894723 29769 server_base.cc:1034] running on GCE node
I20260809 05:21:37.895294 29769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:37.896245 29769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:37.897434 29769 hybrid_clock.cc:648] HybridClock initialized: now 1786252897897381 us; error 43 us; skew 500 ppm
I20260809 05:21:37.897751 29769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:37.899986 29769 webserver.cc:460] Webserver started at http://127.22.1.66:38609/ using document root <none> and password file <none>
I20260809 05:21:37.900676 29769 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:37.900767 29769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:37.901060 29769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:37.903983 29769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "80ffbb39a5474ffda9768fb29d785298"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "9fa7449362ae00c024248bdd209b9de9"
server_key_iv: "88bc4c576c8288a6e0c12c2a7a5ffa0b"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.904791 29769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "80ffbb39a5474ffda9768fb29d785298"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "9fa7449362ae00c024248bdd209b9de9"
server_key_iv: "88bc4c576c8288a6e0c12c2a7a5ffa0b"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.911664 29769 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260809 05:21:37.915293 29788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.916656 29769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:21:37.916812 29769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "80ffbb39a5474ffda9768fb29d785298"
format_stamp: "Formatted at 2026-08-09 05:21:37 on dist-test-slave-rmhg"
server_key: "9fa7449362ae00c024248bdd209b9de9"
server_key_iv: "88bc4c576c8288a6e0c12c2a7a5ffa0b"
server_key_version: "encryptionkey@0"
I20260809 05:21:37.917014 29769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:37.922657 29769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:37.923430 29769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:37.923691 29769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:37.924634 29769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:37.926082 29769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:37.926167 29769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.926271 29769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:37.926337 29769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:37.956698 29769 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:37169
I20260809 05:21:37.956748 29945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:37169 every 8 connection(s)
I20260809 05:21:37.958039 29769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:37.962556 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29769
I20260809 05:21:37.962736 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:37.963183 22533 external_mini_cluster.cc:1428] Setting key b58d6eb948842aea0e0ea1f70ab1b7c3
I20260809 05:21:37.966075 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--builtin_ntp_servers=127.22.1.84:33439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260809 05:21:37.967338 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:37.960588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:48819 (local address 127.22.1.126:46529)
0809 05:21:37.960756 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:37.960761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:37.962475 (+ 1714us) server_negotiation.cc:407] Connection header received
0809 05:21:37.964815 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:37.964819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:37.964881 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:37.964978 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:37.966572 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:37.966578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:37.966582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:37.966602 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:37.966649 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:37.966664 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:37.966776 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:37.966973 (+ 197us) server_negotiation.cc:299] Negotiation successful
0809 05:21:37.967047 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:21:37.968482 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:37.961055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46529 (local address 127.22.1.66:48819)
0809 05:21:37.962273 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:37.962337 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:37.964515 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:37.965277 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:37.965289 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:37.965430 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:37.966191 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:37.966194 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:37.966252 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:37.966362 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:37.966755 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:37.966764 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:37.966771 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 05:21:37.967071 (+ 300us) client_negotiation.cc:240] Negotiation successful
0809 05:21:37.967340 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":335,"threads_started":1}
I20260809 05:21:37.969946 29946 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46529
I20260809 05:21:37.970270 29946 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:37.970988 29946 heartbeater.cc:511] Master 127.22.1.126:46529 requested a full tablet report, sending...
I20260809 05:21:37.972556 29484 ts_manager.cc:194] Registered new tserver with Master: 80ffbb39a5474ffda9768fb29d785298 (127.22.1.66:37169)
W20260809 05:21:38.299953 29949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:38.300264 29949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:38.300385 29949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:38.307778 29949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:38.307904 29949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:38.308058 29949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:38.317625 29949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:38.319000 29949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:38.320881 29949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:38.334353 29960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:38.334324 29963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:38.334290 29961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:38.335207 29949 server_base.cc:1034] running on GCE node
I20260809 05:21:38.335808 29949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:38.336807 29949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:38.338049 29949 hybrid_clock.cc:648] HybridClock initialized: now 1786252898337986 us; error 51 us; skew 500 ppm
I20260809 05:21:38.338402 29949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:38.340883 29949 webserver.cc:460] Webserver started at http://127.22.1.67:40331/ using document root <none> and password file <none>
I20260809 05:21:38.341708 29949 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:38.341812 29949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:38.342185 29949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:38.345330 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "24ace4b08ae54698bfb3ee4cd527478e"
format_stamp: "Formatted at 2026-08-09 05:21:38 on dist-test-slave-rmhg"
server_key: "5be47efd27ff9b409d5a7da237f3a7a8"
server_key_iv: "d8ac52f0f6253967139e68f0909de516"
server_key_version: "encryptionkey@0"
I20260809 05:21:38.346131 29949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "24ace4b08ae54698bfb3ee4cd527478e"
format_stamp: "Formatted at 2026-08-09 05:21:38 on dist-test-slave-rmhg"
server_key: "5be47efd27ff9b409d5a7da237f3a7a8"
server_key_iv: "d8ac52f0f6253967139e68f0909de516"
server_key_version: "encryptionkey@0"
I20260809 05:21:38.351573 29949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:38.355381 29972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:38.356866 29949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 05:21:38.357024 29949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "24ace4b08ae54698bfb3ee4cd527478e"
format_stamp: "Formatted at 2026-08-09 05:21:38 on dist-test-slave-rmhg"
server_key: "5be47efd27ff9b409d5a7da237f3a7a8"
server_key_iv: "d8ac52f0f6253967139e68f0909de516"
server_key_version: "encryptionkey@0"
I20260809 05:21:38.357270 29949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:38.362927 29949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:38.363690 29949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:38.363921 29949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:38.364825 29949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:38.366247 29949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:38.366339 29949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:38.366422 29949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:38.366487 29949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:38.399495 29949 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:45367
I20260809 05:21:38.399549 30141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:45367 every 8 connection(s)
I20260809 05:21:38.400883 29949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:38.401889 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 29949
I20260809 05:21:38.402076 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:38.402504 22533 external_mini_cluster.cc:1428] Setting key 71ce54d70dd5b16ab77057881dd98d82
I20260809 05:21:38.409377 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.403564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:52803 (local address 127.22.1.126:46529)
0809 05:21:38.403736 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.403743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.405568 (+ 1825us) server_negotiation.cc:407] Connection header received
0809 05:21:38.407312 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.407316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.407376 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.407466 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.408757 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.408761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.408764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.408781 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.408814 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.408827 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.408920 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.409065 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.409114 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:21:38.410297 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.404269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46529 (local address 127.22.1.67:52803)
0809 05:21:38.405422 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.405464 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.407067 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.407673 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.407681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.407772 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.408441 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.408443 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.408481 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.408571 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.408897 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.408903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.408908 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.409127 (+ 219us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.409361 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":375,"threads_started":1}
I20260809 05:21:38.411578 30142 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46529
I20260809 05:21:38.411862 30142 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:38.412437 30142 heartbeater.cc:511] Master 127.22.1.126:46529 requested a full tablet report, sending...
I20260809 05:21:38.413825 29484 ts_manager.cc:194] Registered new tserver with Master: 24ace4b08ae54698bfb3ee4cd527478e (127.22.1.67:45367)
I20260809 05:21:38.418174 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:38.424850 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.422472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.22.1.126:46529)
0809 05:21:38.422648 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.422653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.423144 (+ 491us) server_negotiation.cc:407] Connection header received
0809 05:21:38.423279 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.423282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.423335 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.423416 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.424360 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.424364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.424367 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.424379 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.424405 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.424415 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.424515 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.424602 (+ 87us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.424652 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 05:21:38.428757 29484 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:38.435511 29484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:38.453521 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.447420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:37169 (local address 127.0.0.1:41008)
0809 05:21:38.448216 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.448269 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.448415 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.450179 (+ 1764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.450187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.450290 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.451182 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.451186 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.451229 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.451325 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.452720 (+ 1395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.452732 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.452741 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.452917 (+ 176us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.453019 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":355,"threads_started":1}
I20260809 05:21:38.454900 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.447677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.22.1.66:37169)
0809 05:21:38.449544 (+ 1867us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.449553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.449586 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:21:38.449660 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.449668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.449944 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.450844 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.452446 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.452455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.452471 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.452510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.452572 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.452596 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.454277 (+ 1681us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.454363 (+ 86us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.454493 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1671,"thread_start_us":1811,"threads_started":1}
I20260809 05:21:38.459033 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.448770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45367 (local address 127.0.0.1:33822)
0809 05:21:38.453731 (+ 4961us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.453748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.453817 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.456272 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.456278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.456326 (+ 48us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.456906 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.456910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.456929 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.456982 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.458482 (+ 1500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.458489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.458496 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.458632 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.458694 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4874,"mutex_wait_us":230,"threads_started":1}
I20260809 05:21:38.459837 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.449209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.22.1.67:45367)
0809 05:21:38.455698 (+ 6489us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.455707 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.455743 (+ 36us) server_negotiation.cc:407] Connection header received
0809 05:21:38.455814 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.455822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.456078 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.456628 (+ 550us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.458226 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.458236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.458253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.458270 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.458326 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.458345 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.458542 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.459345 (+ 803us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.459458 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6313,"thread_start_us":6129,"threads_started":1}
I20260809 05:21:38.461374 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.448528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43072 (local address 127.22.1.65:38935)
0809 05:21:38.458969 (+ 10441us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.458978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.459010 (+ 32us) server_negotiation.cc:407] Connection header received
0809 05:21:38.459079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.459087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.459314 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.459468 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.460533 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.460539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.460549 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.460569 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.460611 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.460625 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.460745 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.460948 (+ 203us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.461050 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10280,"thread_start_us":10158,"threads_started":1}
I20260809 05:21:38.461405 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.447894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43072)
0809 05:21:38.448936 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.448953 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.449045 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.459485 (+ 10440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.459489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.459511 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.460311 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.460315 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.460332 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.460381 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.460725 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.460730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.460735 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.460871 (+ 136us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.460925 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"thread_start_us":836,"threads_started":1}
I20260809 05:21:38.463529 29852 tablet_service.cc:1467] Processing CreateTablet for tablet cee57425ca884c3283fc8498ee2ebdca (DEFAULT_TABLE table=test-table [id=beddb603a448491faee14d683cbd7b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:38.464179 30040 tablet_service.cc:1467] Processing CreateTablet for tablet cee57425ca884c3283fc8498ee2ebdca (DEFAULT_TABLE table=test-table [id=beddb603a448491faee14d683cbd7b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:38.465207 29852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee57425ca884c3283fc8498ee2ebdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:38.465441 30040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee57425ca884c3283fc8498ee2ebdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:38.465821 29668 tablet_service.cc:1467] Processing CreateTablet for tablet cee57425ca884c3283fc8498ee2ebdca (DEFAULT_TABLE table=test-table [id=beddb603a448491faee14d683cbd7b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:38.467474 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee57425ca884c3283fc8498ee2ebdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:38.477427 30167 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Bootstrap starting.
I20260809 05:21:38.480650 30167 tablet_bootstrap.cc:654] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:38.481556 30168 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Bootstrap starting.
I20260809 05:21:38.481982 30167 log.cc:824] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:38.483135 30169 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Bootstrap starting.
I20260809 05:21:38.485136 30168 tablet_bootstrap.cc:654] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:38.486455 30169 tablet_bootstrap.cc:654] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:38.486549 30168 log.cc:824] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:38.487020 30167 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: No bootstrap required, opened a new log
I20260809 05:21:38.487355 30167 ts_tablet_manager.cc:1397] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 05:21:38.487555 30169 log.cc:824] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:38.489655 30168 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: No bootstrap required, opened a new log
I20260809 05:21:38.489912 30168 ts_tablet_manager.cc:1397] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 05:21:38.490098 30169 tablet_bootstrap.cc:492] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: No bootstrap required, opened a new log
I20260809 05:21:38.490334 30169 ts_tablet_manager.cc:1397] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 05:21:38.497400 30169 raft_consensus.cc:348] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.497419 30167 raft_consensus.cc:348] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.497745 30169 raft_consensus.cc:374] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:38.497828 30169 raft_consensus.cc:729] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f785d6797194f42abf7770ec9a6b539, State: Initialized, Role: FOLLOWER
I20260809 05:21:38.497879 30167 raft_consensus.cc:374] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:38.497994 30167 raft_consensus.cc:729] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ace4b08ae54698bfb3ee4cd527478e, State: Initialized, Role: FOLLOWER
I20260809 05:21:38.498420 30169 consensus_queue.cc:260] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.498684 30167 consensus_queue.cc:260] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.499433 29764 heartbeater.cc:503] Master 127.22.1.126:46529 was elected leader, sending a full tablet report...
I20260809 05:21:38.499743 30168 raft_consensus.cc:348] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.500102 30169 ts_tablet_manager.cc:1428] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260809 05:21:38.500161 30168 raft_consensus.cc:374] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:38.500242 30168 raft_consensus.cc:729] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80ffbb39a5474ffda9768fb29d785298, State: Initialized, Role: FOLLOWER
I20260809 05:21:38.501168 30142 heartbeater.cc:503] Master 127.22.1.126:46529 was elected leader, sending a full tablet report...
I20260809 05:21:38.500967 30168 consensus_queue.cc:260] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.501823 30167 ts_tablet_manager.cc:1428] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260809 05:21:38.502666 29946 heartbeater.cc:503] Master 127.22.1.126:46529 was elected leader, sending a full tablet report...
I20260809 05:21:38.503254 30168 ts_tablet_manager.cc:1428] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260809 05:21:38.524194 29766 tablet.cc:2369] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:38.641557 30183 raft_consensus.cc:482] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:38.641914 30183 raft_consensus.cc:504] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.643683 30183 leader_election.cc:288] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24ace4b08ae54698bfb3ee4cd527478e (127.22.1.67:45367), 6f785d6797194f42abf7770ec9a6b539 (127.22.1.65:38935)
I20260809 05:21:38.647189 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.644062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:55777 (local address 127.22.1.67:45367)
0809 05:21:38.644265 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.644273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.645210 (+ 937us) server_negotiation.cc:407] Connection header received
0809 05:21:38.645445 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.645450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.645519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.645612 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.646629 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.646634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.646638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.646655 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.646689 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.646704 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.646814 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.646868 (+ 54us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.646929 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:21:38.647207 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.644183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57611 (local address 127.22.1.65:38935)
0809 05:21:38.644362 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.644369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.645102 (+ 733us) server_negotiation.cc:407] Connection header received
0809 05:21:38.645398 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.645404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.645471 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.645556 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.646589 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.646594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.646598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.646619 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.646660 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.646676 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.646802 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.646873 (+ 71us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.646929 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:21:38.647527 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.644062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.22.1.66:57611)
0809 05:21:38.644987 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.645006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.645247 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.645604 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.645609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.645633 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.646341 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.646344 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.646363 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.646419 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.646771 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.646776 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.646780 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.646934 (+ 154us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.646987 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":877,"threads_started":1}
I20260809 05:21:38.647580 30185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.644119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:45367 (local address 127.22.1.66:55777)
0809 05:21:38.645089 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.645107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:38.645311 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:38.645742 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:38.645750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:38.645768 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:38.646439 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:21:38.646442 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:38.646454 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:21:38.646487 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:38.646781 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:38.646784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:38.646788 (+ 4us) client_negotiation.cc:769] Sending connection context
0809 05:21:38.646928 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 05:21:38.646997 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":3,"spinlock_wait_cycles":24960,"thread_start_us":474,"threads_started":1}
I20260809 05:21:38.648396 29696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee57425ca884c3283fc8498ee2ebdca" candidate_uuid: "80ffbb39a5474ffda9768fb29d785298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f785d6797194f42abf7770ec9a6b539" is_pre_election: true
I20260809 05:21:38.648447 30072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee57425ca884c3283fc8498ee2ebdca" candidate_uuid: "80ffbb39a5474ffda9768fb29d785298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ace4b08ae54698bfb3ee4cd527478e" is_pre_election: true
I20260809 05:21:38.648881 30072 raft_consensus.cc:2454] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ffbb39a5474ffda9768fb29d785298 in term 0.
I20260809 05:21:38.648896 29696 raft_consensus.cc:2454] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80ffbb39a5474ffda9768fb29d785298 in term 0.
I20260809 05:21:38.649542 29794 leader_election.cc:302] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f785d6797194f42abf7770ec9a6b539, 80ffbb39a5474ffda9768fb29d785298; no voters:
I20260809 05:21:38.649927 30183 raft_consensus.cc:2781] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:38.650069 30183 raft_consensus.cc:482] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:38.650164 30183 raft_consensus.cc:3037] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:38.652307 30183 raft_consensus.cc:504] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
W20260809 05:21:38.652634 30143 tablet.cc:2369] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:38.652899 30183 leader_election.cc:288] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [CANDIDATE]: Term 1 election: Requested vote from peers 24ace4b08ae54698bfb3ee4cd527478e (127.22.1.67:45367), 6f785d6797194f42abf7770ec9a6b539 (127.22.1.65:38935)
I20260809 05:21:38.653359 30072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee57425ca884c3283fc8498ee2ebdca" candidate_uuid: "80ffbb39a5474ffda9768fb29d785298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ace4b08ae54698bfb3ee4cd527478e"
I20260809 05:21:38.653398 29696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee57425ca884c3283fc8498ee2ebdca" candidate_uuid: "80ffbb39a5474ffda9768fb29d785298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f785d6797194f42abf7770ec9a6b539"
I20260809 05:21:38.653554 30072 raft_consensus.cc:3037] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:38.653614 29696 raft_consensus.cc:3037] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:38.655654 30072 raft_consensus.cc:2454] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ffbb39a5474ffda9768fb29d785298 in term 1.
I20260809 05:21:38.655766 29696 raft_consensus.cc:2454] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80ffbb39a5474ffda9768fb29d785298 in term 1.
I20260809 05:21:38.656092 29789 leader_election.cc:302] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ace4b08ae54698bfb3ee4cd527478e, 80ffbb39a5474ffda9768fb29d785298; no voters:
I20260809 05:21:38.656414 30183 raft_consensus.cc:2781] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:38.656818 30183 raft_consensus.cc:686] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 1 LEADER]: Becoming Leader. State: Replica: 80ffbb39a5474ffda9768fb29d785298, State: Running, Role: LEADER
I20260809 05:21:38.657317 30183 consensus_queue.cc:237] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } }
I20260809 05:21:38.662132 29482 catalog_manager.cc:5495] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80ffbb39a5474ffda9768fb29d785298 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "80ffbb39a5474ffda9768fb29d785298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80ffbb39a5474ffda9768fb29d785298" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 37169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:38.700531 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.698569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.22.1.66:37169)
0809 05:21:38.698884 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.698889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.698906 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:21:38.699019 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.699023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.699100 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.699178 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.699987 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.699992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.699995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.700010 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.700043 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.700055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.700149 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.700216 (+ 67us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.700262 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260809 05:21:38.708631 29696 raft_consensus.cc:1261] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539 [term 1 FOLLOWER]: Refusing update from remote peer 80ffbb39a5474ffda9768fb29d785298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:38.708662 30072 raft_consensus.cc:1261] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 1 FOLLOWER]: Refusing update from remote peer 80ffbb39a5474ffda9768fb29d785298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:38.709626 30187 consensus_queue.cc:1035] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f785d6797194f42abf7770ec9a6b539" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260809 05:21:38.709988 29947 tablet.cc:2369] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:38.710031 30183 consensus_queue.cc:1035] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ace4b08ae54698bfb3ee4cd527478e" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:38.722397 30194 mvcc.cc:205] Tried to move back new op lower bound from 7316491873101221888 to 7316491872904192000. Current Snapshot: MvccSnapshot[applied={T|T < 7316491873101221888}]
I20260809 05:21:38.735155 29484 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:38.735399 29484 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:38.738847 29484 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15cd2325123f43e28f1639c5e22e1bad: Sending DeleteTablet for 3 replicas of tablet cee57425ca884c3283fc8498ee2ebdca
I20260809 05:21:38.740121 29668 tablet_service.cc:1514] Processing DeleteTablet for tablet cee57425ca884c3283fc8498ee2ebdca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:38 UTC) from {username='slave'} at 127.0.0.1:43072
I20260809 05:21:38.740384 29852 tablet_service.cc:1514] Processing DeleteTablet for tablet cee57425ca884c3283fc8498ee2ebdca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:38 UTC) from {username='slave'} at 127.0.0.1:41008
I20260809 05:21:38.740406 30040 tablet_service.cc:1514] Processing DeleteTablet for tablet cee57425ca884c3283fc8498ee2ebdca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:38 UTC) from {username='slave'} at 127.0.0.1:33822
I20260809 05:21:38.741454 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29572
I20260809 05:21:38.756410 30206 tablet_replica.cc:330] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: stopping tablet replica
I20260809 05:21:38.756567 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.744398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.22.1.65:38935)
0809 05:21:38.744629 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.744635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.753486 (+ 8851us) server_negotiation.cc:407] Connection header received
0809 05:21:38.754055 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.754062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.754194 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.754278 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.755575 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.755582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.755588 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.755614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.755660 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.755676 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.755783 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.756033 (+ 250us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.756108 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 05:21:38.757000 30206 raft_consensus.cc:2229] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:38.757623 30206 raft_consensus.cc:2258] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:38.757997 29757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:38.763620 30206 ts_tablet_manager.cc:1905] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:38.770256 30205 tablet_replica.cc:330] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: stopping tablet replica
I20260809 05:21:38.770790 30205 raft_consensus.cc:2229] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:38.771538 30205 raft_consensus.cc:2258] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:38.773828 30205 ts_tablet_manager.cc:1905] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:38.773932 30206 ts_tablet_manager.cc:1918] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:38.774121 30206 log.cc:1196] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/cee57425ca884c3283fc8498ee2ebdca
I20260809 05:21:38.774664 30206 ts_tablet_manager.cc:1939] T cee57425ca884c3283fc8498ee2ebdca P 24ace4b08ae54698bfb3ee4cd527478e: Deleting consensus metadata
I20260809 05:21:38.776582 29460 catalog_manager.cc:4841] TS 24ace4b08ae54698bfb3ee4cd527478e (127.22.1.67:45367): tablet cee57425ca884c3283fc8498ee2ebdca (table test-table [id=beddb603a448491faee14d683cbd7b1e]) successfully deleted
I20260809 05:21:38.780015 30205 ts_tablet_manager.cc:1918] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:38.780185 30205 log.cc:1196] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/cee57425ca884c3283fc8498ee2ebdca
I20260809 05:21:38.780652 30205 ts_tablet_manager.cc:1939] T cee57425ca884c3283fc8498ee2ebdca P 80ffbb39a5474ffda9768fb29d785298: Deleting consensus metadata
I20260809 05:21:38.782147 29460 catalog_manager.cc:4841] TS 80ffbb39a5474ffda9768fb29d785298 (127.22.1.66:37169): tablet cee57425ca884c3283fc8498ee2ebdca (table test-table [id=beddb603a448491faee14d683cbd7b1e]) successfully deleted
I20260809 05:21:38.883448 30204 tablet_replica.cc:330] T cee57425ca884c3283fc8498ee2ebdca P 6f785d6797194f42abf7770ec9a6b539: stopping tablet replica
I20260809 05:21:38.883605 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29572
W20260809 05:21:38.912029 29465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:38935 (error 108)
I20260809 05:21:38.912245 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29769
I20260809 05:21:38.914060 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.913595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43086)
0809 05:21:38.913763 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.913879 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 05:21:38.914410 29465 catalog_manager.cc:4584] TS 6f785d6797194f42abf7770ec9a6b539 (127.22.1.65:38935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cee57425ca884c3283fc8498ee2ebdca: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
I20260809 05:21:38.915643 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.913194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.22.1.66:37169)
0809 05:21:38.913366 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:38.913371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:38.913644 (+ 273us) server_negotiation.cc:407] Connection header received
0809 05:21:38.913736 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:38.913742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:38.913811 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:38.913911 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:38.914929 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:38.914935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:38.914940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:38.914959 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:38.915002 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:38.915019 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:38.915152 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:38.915271 (+ 119us) server_negotiation.cc:299] Negotiation successful
0809 05:21:38.915339 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:21:38.916415 29939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:38.968365 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.967781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43102)
0809 05:21:38.968024 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.968167 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 05:21:38.969565 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:38.969131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43114)
0809 05:21:38.969315 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:38.969415 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 05:21:39.027171 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29769
I20260809 05:21:39.030027 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.029572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43130)
0809 05:21:39.029767 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:39.029864 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 05:21:39.031122 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.030780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43134)
0809 05:21:39.030918 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:39.031001 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 05:21:39.049788 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29949
I20260809 05:21:39.052491 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.050608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33834 (local address 127.22.1.67:45367)
0809 05:21:39.050776 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:39.050781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:39.050800 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:21:39.050973 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:39.050977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:39.051072 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:39.051161 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:39.052000 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:39.052004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:39.052007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:39.052020 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:39.052051 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:39.052063 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:39.052169 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:39.052209 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 05:21:39.052257 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:21:39.053134 30135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:39.111966 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.111340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43142)
0809 05:21:39.111632 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:39.111751 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260809 05:21:39.113108 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.112643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:38935 (local address 127.0.0.1:43144)
0809 05:21:39.112841 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:39.112933 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260809 05:21:39.140429 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29949
I20260809 05:21:39.162894 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 29440
I20260809 05:21:39.164223 29528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:39.270239 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 29440
2026-08-09T05:21:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2783 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260809 05:21:39.311424 22533 mini_oidc.cc:152] Starting JWKS server
I20260809 05:21:39.314483 22533 webserver.cc:460] Webserver started at https://127.0.0.1:32889/ using document root <none> and password file <none>
I20260809 05:21:39.314813 22533 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 05:21:39.315516 22533 webserver.cc:460] Webserver started at http://127.0.0.1:34363/ using document root <none> and password file <none>
2026-08-09T05:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:39Z Disabled control of system clock
I20260809 05:21:39.347035 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36459
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:36459
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32889/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/testchainca.cert with env {}
W20260809 05:21:39.678126 30224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:39.678421 30224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:39.678540 30224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:39.685727 30224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:39.685834 30224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:39.685892 30224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:39.685977 30224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:39.686038 30224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 05:21:39.686122 30224 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 05:21:39.686169 30224 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32889/jwks.json
I20260809 05:21:39.696102 30224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:36459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32889/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:39.697413 30224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:39.698767 30224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:39.710259 30234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:39.710299 30235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:39.710320 30237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:39.711043 30224 server_base.cc:1034] running on GCE node
I20260809 05:21:39.711792 30224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:39.713248 30224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:39.714495 30224 hybrid_clock.cc:648] HybridClock initialized: now 1786252899714427 us; error 61 us; skew 500 ppm
I20260809 05:21:39.714844 30224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:39.717299 30224 webserver.cc:460] Webserver started at http://127.22.1.126:39741/ using document root <none> and password file <none>
I20260809 05:21:39.717971 30224 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:39.718081 30224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:39.718387 30224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:39.721307 30224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "a1b45d2c26e3477c8f5192b6e2042c8d"
format_stamp: "Formatted at 2026-08-09 05:21:39 on dist-test-slave-rmhg"
server_key: "56529e8c0498a6e2e94e83135f3deceb"
server_key_iv: "33128eaa462da2fdff2e1902e6d329ee"
server_key_version: "encryptionkey@0"
I20260809 05:21:39.722083 30224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "a1b45d2c26e3477c8f5192b6e2042c8d"
format_stamp: "Formatted at 2026-08-09 05:21:39 on dist-test-slave-rmhg"
server_key: "56529e8c0498a6e2e94e83135f3deceb"
server_key_iv: "33128eaa462da2fdff2e1902e6d329ee"
server_key_version: "encryptionkey@0"
I20260809 05:21:39.727255 30224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:39.730775 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:39.732080 30224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:21:39.732239 30224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "a1b45d2c26e3477c8f5192b6e2042c8d"
format_stamp: "Formatted at 2026-08-09 05:21:39 on dist-test-slave-rmhg"
server_key: "56529e8c0498a6e2e94e83135f3deceb"
server_key_iv: "33128eaa462da2fdff2e1902e6d329ee"
server_key_version: "encryptionkey@0"
I20260809 05:21:39.732431 30224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:39.751569 30224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:39.752530 30224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:39.752820 30224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:39.770920 30224 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:36459
I20260809 05:21:39.770958 30327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:36459 every 8 connection(s)
I20260809 05:21:39.772284 30224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:39.776283 30328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:39.776643 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 30224
I20260809 05:21:39.776809 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:39.777489 22533 external_mini_cluster.cc:1428] Setting key 7c78b4a62eb28cc8c364a9397517c6c1
I20260809 05:21:39.790783 30328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d: Bootstrap starting.
I20260809 05:21:39.793582 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.779480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.22.1.126:36459)
0809 05:21:39.780623 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:39.780638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:39.780701 (+ 63us) server_negotiation.cc:407] Connection header received
0809 05:21:39.783008 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:39.783040 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:39.783703 (+ 663us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:39.784192 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:39.784846 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.786781 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.788142 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.789202 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.789373 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:39.790610 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:39.790621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:39.790640 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:39.791371 (+ 731us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:39.791474 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:39.791502 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:39.791604 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:39.791849 (+ 245us) server_negotiation.cc:299] Negotiation successful
0809 05:21:39.792200 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":342,"threads_started":1}
I20260809 05:21:39.794724 30328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:39.796072 30328 log.cc:824] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:39.797782 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:21:39.799026 30328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d: No bootstrap required, opened a new log
I20260809 05:21:39.806964 30328 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } }
I20260809 05:21:39.807404 30328 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:39.807515 30328 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b45d2c26e3477c8f5192b6e2042c8d, State: Initialized, Role: FOLLOWER
I20260809 05:21:39.808118 30328 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } }
I20260809 05:21:39.808327 30328 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:39.808454 30328 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:39.808615 30328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:39.810828 30328 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } }
I20260809 05:21:39.811322 30328 leader_election.cc:302] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b45d2c26e3477c8f5192b6e2042c8d; no voters:
I20260809 05:21:39.811882 30328 leader_election.cc:288] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:39.812023 30336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:39.812685 30336 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [term 1 LEADER]: Becoming Leader. State: Replica: a1b45d2c26e3477c8f5192b6e2042c8d, State: Running, Role: LEADER
I20260809 05:21:39.813232 30336 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } }
I20260809 05:21:39.814109 30328 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:39.816051 30339 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } } }
I20260809 05:21:39.816493 30339 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:39.816077 30340 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b45d2c26e3477c8f5192b6e2042c8d. Latest consensus state: current_term: 1 leader_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b45d2c26e3477c8f5192b6e2042c8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36459 } } }
I20260809 05:21:39.817359 30340 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:39.818742 30348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:39.823221 30348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:39.831652 30348 catalog_manager.cc:1347] Generated new cluster ID: 3caa60827b894d10880351ce174f21d5
I20260809 05:21:39.831770 30348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:39.855651 30348 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:39.857610 30348 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:39.865702 30348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b45d2c26e3477c8f5192b6e2042c8d: Generated new TSK 0
I20260809 05:21:39.866578 30348 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 05:21:39.915531 30372 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260809 05:21:39.915695 30372 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 05:21:39.917028 30333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:39.894458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.22.1.126:36459)
0809 05:21:39.894662 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:39.894669 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:39.895038 (+ 369us) server_negotiation.cc:407] Connection header received
0809 05:21:39.895424 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:39.895430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:39.895546 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:39.895717 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0809 05:21:39.896235 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.897427 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.898888 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.899586 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.899711 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:39.900215 (+ 504us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 05:21:39.916320 (+ 16105us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 05:21:39.916708 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 05:21:39.917179 30333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260809 05:21:39.918754 30371 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:39.894272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:36459 (local address 127.0.0.1:49094)
0809 05:21:39.894925 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:39.894945 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:39.895101 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:39.895732 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:39.895735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:39.895745 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0809 05:21:39.896076 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:39.896084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.897580 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.897585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:39.898748 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:39.898760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:39.899739 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:39.899743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:39.899862 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:39.899925 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 05:21:39.916636 (+ 16711us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0809 05:21:39.916921 (+ 285us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.1.126:36459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.22.1.126:36459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260809 05:21:39.920593 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 30224
I20260809 05:21:39.921712 30323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:40.016710 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 30224
2026-08-09T05:21:40Z chronyd exiting
I20260809 05:21:40.075649 22533 test_util.cc:175] -----------------------------------------------
I20260809 05:21:40.075780 22533 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786252853919692-22533-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (769 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260809 05:21:40.080493 22533 mini_oidc.cc:152] Starting JWKS server
I20260809 05:21:40.082387 22533 webserver.cc:460] Webserver started at https://127.0.0.1:39601/ using document root <none> and password file <none>
I20260809 05:21:40.082657 22533 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 05:21:40.083273 22533 webserver.cc:460] Webserver started at http://127.0.0.1:41711/ using document root <none> and password file <none>
2026-08-09T05:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:40Z Disabled control of system clock
I20260809 05:21:40.114410 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39909
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:40503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:39909
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39601/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/testchainca.cert with env {}
W20260809 05:21:40.445482 30381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:40.445811 30381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:40.445948 30381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:40.453363 30381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:40.453510 30381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:40.453573 30381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:40.453656 30381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:40.453715 30381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 05:21:40.453804 30381 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 05:21:40.453859 30381 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39601/jwks.json
I20260809 05:21:40.464406 30381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:39909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39601/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:40.465899 30381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:40.467250 30381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:40.479323 30389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:40.479439 30392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:40.479295 30388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:40.480310 30381 server_base.cc:1034] running on GCE node
I20260809 05:21:40.481201 30381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:40.482746 30381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:40.483966 30381 hybrid_clock.cc:648] HybridClock initialized: now 1786252900483907 us; error 49 us; skew 500 ppm
I20260809 05:21:40.484328 30381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:40.491246 30381 webserver.cc:460] Webserver started at http://127.22.1.126:38569/ using document root <none> and password file <none>
I20260809 05:21:40.492005 30381 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:40.492134 30381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:40.492471 30381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:40.495558 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "824d185a0f6849bbb201c94e2174d355"
format_stamp: "Formatted at 2026-08-09 05:21:40 on dist-test-slave-rmhg"
server_key: "8c18e0748676e994343930b35e68369f"
server_key_iv: "480805b4c49e23eef05e6f18884b330a"
server_key_version: "encryptionkey@0"
I20260809 05:21:40.496347 30381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "824d185a0f6849bbb201c94e2174d355"
format_stamp: "Formatted at 2026-08-09 05:21:40 on dist-test-slave-rmhg"
server_key: "8c18e0748676e994343930b35e68369f"
server_key_iv: "480805b4c49e23eef05e6f18884b330a"
server_key_version: "encryptionkey@0"
I20260809 05:21:40.501645 30381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:40.505141 30400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:40.506517 30381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:40.506683 30381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "824d185a0f6849bbb201c94e2174d355"
format_stamp: "Formatted at 2026-08-09 05:21:40 on dist-test-slave-rmhg"
server_key: "8c18e0748676e994343930b35e68369f"
server_key_iv: "480805b4c49e23eef05e6f18884b330a"
server_key_version: "encryptionkey@0"
I20260809 05:21:40.506891 30381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:40.542608 30381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:40.543531 30381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:40.543826 30381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:40.561496 30381 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:39909
I20260809 05:21:40.561525 30477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:39909 every 8 connection(s)
I20260809 05:21:40.562904 30381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:40.565032 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 30381
I20260809 05:21:40.565277 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:40.566094 22533 external_mini_cluster.cc:1428] Setting key a632ca5eac5cc3be1e131a9974421cb5
I20260809 05:21:40.567615 30479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:40.580590 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:40.568186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52356 (local address 127.22.1.126:39909)
0809 05:21:40.569437 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:40.569448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:40.569553 (+ 105us) server_negotiation.cc:407] Connection header received
0809 05:21:40.571235 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:40.571259 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:40.571762 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:40.572142 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:40.572761 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.574144 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.575966 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.576631 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.576768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:40.578134 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:40.578142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:40.578160 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:40.578752 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:40.578835 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:40.578857 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:40.578987 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:40.579207 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 05:21:40.579579 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":402,"threads_started":1}
I20260809 05:21:40.581645 30479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355: Bootstrap starting.
I20260809 05:21:40.584549 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:21:40.585124 30479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:40.586432 30479 log.cc:824] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:40.589690 30479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355: No bootstrap required, opened a new log
I20260809 05:21:40.597355 30479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } }
I20260809 05:21:40.597734 30479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:40.597816 30479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824d185a0f6849bbb201c94e2174d355, State: Initialized, Role: FOLLOWER
I20260809 05:21:40.598320 30479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } }
I20260809 05:21:40.598565 30479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:40.598668 30479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:40.598795 30479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:40.600951 30479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } }
I20260809 05:21:40.601501 30479 leader_election.cc:302] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824d185a0f6849bbb201c94e2174d355; no voters:
I20260809 05:21:40.602110 30479 leader_election.cc:288] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:40.602244 30488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:40.602937 30488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [term 1 LEADER]: Becoming Leader. State: Replica: 824d185a0f6849bbb201c94e2174d355, State: Running, Role: LEADER
I20260809 05:21:40.603464 30488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } }
I20260809 05:21:40.604247 30479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:40.606382 30489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824d185a0f6849bbb201c94e2174d355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } } }
I20260809 05:21:40.606777 30489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:40.606350 30490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824d185a0f6849bbb201c94e2174d355. Latest consensus state: current_term: 1 leader_uuid: "824d185a0f6849bbb201c94e2174d355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824d185a0f6849bbb201c94e2174d355" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39909 } } }
I20260809 05:21:40.607020 30490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:40.608219 30497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:40.617511 30497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:40.625906 30497 catalog_manager.cc:1347] Generated new cluster ID: d36073e94bc5453e90547df2006ed8b2
I20260809 05:21:40.626034 30497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:40.637622 30497 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:40.638958 30497 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:40.650806 30497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824d185a0f6849bbb201c94e2174d355: Generated new TSK 0
I20260809 05:21:40.652070 30497 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260809 05:21:40.700028 30527 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260809 05:21:40.700193 30527 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 05:21:40.701195 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:40.673435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.22.1.126:39909)
0809 05:21:40.673697 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:40.673703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:40.674267 (+ 564us) server_negotiation.cc:407] Connection header received
0809 05:21:40.674497 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:40.674502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:40.674610 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:40.674739 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0809 05:21:40.675276 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.676009 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.678067 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.678536 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.678633 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:40.679177 (+ 544us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 05:21:40.700591 (+ 21414us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 05:21:40.700916 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":104}
W20260809 05:21:40.701345 30482 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260809 05:21:40.701627 30526 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:40.673193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:39909 (local address 127.0.0.1:52360)
0809 05:21:40.674152 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:40.674174 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:40.674337 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:40.674779 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:40.674784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:40.674798 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0809 05:21:40.675125 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:40.675138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.676148 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.676154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:40.677930 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:40.677947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:40.678699 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:40.678705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:40.678903 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:40.678952 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 05:21:40.700869 (+ 21917us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0809 05:21:40.701212 (+ 343us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.1.126:39909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39601/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":640,"threads_started":1}
I20260809 05:21:40.703101 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 30381
I20260809 05:21:40.704252 30472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:40.798075 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 30381
2026-08-09T05:21:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (782 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260809 05:21:40.861853 22533 mini_oidc.cc:152] Starting JWKS server
I20260809 05:21:40.863788 22533 webserver.cc:460] Webserver started at https://127.0.0.1:33131/ using document root <none> and password file <none>
I20260809 05:21:40.864023 22533 mini_oidc.cc:189] Starting OIDC Discovery server
I20260809 05:21:40.864696 22533 webserver.cc:460] Webserver started at http://127.0.0.1:37655/ using document root <none> and password file <none>
2026-08-09T05:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:40Z Disabled control of system clock
I20260809 05:21:40.896003 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39005
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:43859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:39005
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 05:21:41.242844 30539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:41.243162 30539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:41.243290 30539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:41.250636 30539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:41.250757 30539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:41.250819 30539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:41.250905 30539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:41.250962 30539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 05:21:41.251048 30539 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 05:21:41.251097 30539 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33131/jwks.json
I20260809 05:21:41.260500 30539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:43859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:39005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:41.262019 30539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:41.263916 30539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:41.276031 30549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:41.276026 30554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:41.276021 30552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:41.276801 30539 server_base.cc:1034] running on GCE node
I20260809 05:21:41.277637 30539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:41.279057 30539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:41.280261 30539 hybrid_clock.cc:648] HybridClock initialized: now 1786252901280205 us; error 52 us; skew 500 ppm
I20260809 05:21:41.280588 30539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:41.282915 30539 webserver.cc:460] Webserver started at http://127.22.1.126:45283/ using document root <none> and password file <none>
I20260809 05:21:41.283608 30539 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:41.283699 30539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:41.283995 30539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:41.287034 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "ded36f8b1ebc4b78821d36d533b7778e"
format_stamp: "Formatted at 2026-08-09 05:21:41 on dist-test-slave-rmhg"
server_key: "625efaba66947d7afb2476d9879167b3"
server_key_iv: "760b6a33f6b281aae8ef0499f8e7b4d2"
server_key_version: "encryptionkey@0"
I20260809 05:21:41.287835 30539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "ded36f8b1ebc4b78821d36d533b7778e"
format_stamp: "Formatted at 2026-08-09 05:21:41 on dist-test-slave-rmhg"
server_key: "625efaba66947d7afb2476d9879167b3"
server_key_iv: "760b6a33f6b281aae8ef0499f8e7b4d2"
server_key_version: "encryptionkey@0"
I20260809 05:21:41.293200 30539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:21:41.296824 30566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:41.298233 30539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:41.298390 30539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "ded36f8b1ebc4b78821d36d533b7778e"
format_stamp: "Formatted at 2026-08-09 05:21:41 on dist-test-slave-rmhg"
server_key: "625efaba66947d7afb2476d9879167b3"
server_key_iv: "760b6a33f6b281aae8ef0499f8e7b4d2"
server_key_version: "encryptionkey@0"
I20260809 05:21:41.298578 30539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:41.317895 30539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:41.318780 30539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:41.319049 30539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:41.337415 30539 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:39005
I20260809 05:21:41.337433 30636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:39005 every 8 connection(s)
I20260809 05:21:41.338820 30539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:41.342701 30637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:41.347038 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 30539
I20260809 05:21:41.347200 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:41.347834 22533 external_mini_cluster.cc:1428] Setting key 4874d0904cbe5750d10e5cf3adbb4d99
I20260809 05:21:41.354848 30637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e: Bootstrap starting.
I20260809 05:21:41.358507 30637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:41.359885 30637 log.cc:824] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:41.363564 30637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e: No bootstrap required, opened a new log
I20260809 05:21:41.363915 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:41.349786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.22.1.126:39005)
0809 05:21:41.350863 (+ 1077us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:41.350882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:41.350939 (+ 57us) server_negotiation.cc:407] Connection header received
0809 05:21:41.353329 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:41.353361 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:41.353954 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:41.354410 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:41.354954 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.356911 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.358398 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.359409 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.359603 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:41.360910 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:41.360919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:41.360952 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:41.361759 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:41.361873 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:41.361903 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:41.362034 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:41.362272 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 05:21:41.362637 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
I20260809 05:21:41.368124 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:21:41.373718 30637 raft_consensus.cc:348] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } }
I20260809 05:21:41.374346 30637 raft_consensus.cc:374] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:41.374511 30637 raft_consensus.cc:729] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded36f8b1ebc4b78821d36d533b7778e, State: Initialized, Role: FOLLOWER
I20260809 05:21:41.375346 30637 consensus_queue.cc:260] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } }
I20260809 05:21:41.375689 30637 raft_consensus.cc:388] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:41.375883 30637 raft_consensus.cc:482] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:41.376168 30637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:41.379552 30637 raft_consensus.cc:504] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } }
I20260809 05:21:41.380072 30637 leader_election.cc:302] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ded36f8b1ebc4b78821d36d533b7778e; no voters:
I20260809 05:21:41.380661 30637 leader_election.cc:288] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:41.380787 30644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:41.381556 30644 raft_consensus.cc:686] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [term 1 LEADER]: Becoming Leader. State: Replica: ded36f8b1ebc4b78821d36d533b7778e, State: Running, Role: LEADER
I20260809 05:21:41.382099 30644 consensus_queue.cc:237] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } }
I20260809 05:21:41.382784 30637 sys_catalog.cc:564] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:41.384872 30647 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ded36f8b1ebc4b78821d36d533b7778e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } } }
I20260809 05:21:41.384881 30648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ded36f8b1ebc4b78821d36d533b7778e. Latest consensus state: current_term: 1 leader_uuid: "ded36f8b1ebc4b78821d36d533b7778e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ded36f8b1ebc4b78821d36d533b7778e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39005 } } }
I20260809 05:21:41.385346 30647 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:41.385346 30648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:41.386866 30654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:41.391722 30654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:41.401286 30654 catalog_manager.cc:1347] Generated new cluster ID: 6060229457d845de90111b94138544a8
I20260809 05:21:41.401464 30654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:41.431443 30654 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:41.432782 30654 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:41.441951 30654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ded36f8b1ebc4b78821d36d533b7778e: Generated new TSK 0
I20260809 05:21:41.442975 30654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:21:41.479869 30642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:41.470958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.22.1.126:39005)
0809 05:21:41.471183 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:41.471189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:41.471641 (+ 452us) server_negotiation.cc:407] Connection header received
0809 05:21:41.471851 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:41.471856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:41.471956 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:41.472148 (+ 192us) server_negotiation.cc:226] Negotiated authn=JWT
0809 05:21:41.472537 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.473764 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.475229 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.475916 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.476066 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:41.476619 (+ 553us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 05:21:41.479186 (+ 2567us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 05:21:41.479552 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 05:21:41.480026 30642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260809 05:21:41.481478 30682 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:41.470899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:39005 (local address 127.0.0.1:38244)
0809 05:21:41.471537 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:41.471553 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:41.471684 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:41.472142 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:41.472145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:41.472154 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0809 05:21:41.472394 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:41.472402 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.473972 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.473977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:41.475072 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:41.475086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:41.476129 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:41.476134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:41.476279 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:41.476318 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 05:21:41.479427 (+ 3109us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0809 05:21:41.481028 (+ 1601us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.1.126:39005: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":1251,"thread_start_us":343,"threads_started":1}
I20260809 05:21:41.483134 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 30539
I20260809 05:21:41.484486 30629 generic_service.cc:196] Checking for leaks (request via RPC)
E20260809 05:21:41.494141 30683 webserver.cc:533] Webserver: sslize failed:
E20260809 05:21:41.494285 30683 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260809 05:21:41.580874 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 30539
2026-08-09T05:21:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (781 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-09T05:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:41Z Disabled control of system clock
I20260809 05:21:41.675323 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:42045
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:33089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:42045
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/testchainca.cert with env {}
W20260809 05:21:42.007755 30690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:42.008068 30690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:42.008155 30690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:42.015364 30690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:42.015488 30690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:42.015532 30690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:42.015589 30690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:42.015632 30690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260809 05:21:42.015692 30690 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260809 05:21:42.015733 30690 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260809 05:21:42.026252 30690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:42045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:42.027630 30690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:42.028949 30690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:42.040657 30699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:42.040621 30701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:42.040702 30705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:42.042045 30690 server_base.cc:1034] running on GCE node
I20260809 05:21:42.042906 30690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:42.044409 30690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:42.045647 30690 hybrid_clock.cc:648] HybridClock initialized: now 1786252902045596 us; error 44 us; skew 500 ppm
I20260809 05:21:42.046020 30690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:21:42.048415 30690 webserver.cc:460] Webserver started at http://127.22.1.126:37805/ using document root <none> and password file <none>
I20260809 05:21:42.049217 30690 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:42.049336 30690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:42.049659 30690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:42.052623 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "589df682b1d24699936a0e75c71ded1f"
format_stamp: "Formatted at 2026-08-09 05:21:42 on dist-test-slave-rmhg"
server_key: "6792583b4090741f681a9dc0da0b80a9"
server_key_iv: "2e90d3677c77f3cda13a84394ac8ea44"
server_key_version: "encryptionkey@0"
I20260809 05:21:42.053470 30690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "589df682b1d24699936a0e75c71ded1f"
format_stamp: "Formatted at 2026-08-09 05:21:42 on dist-test-slave-rmhg"
server_key: "6792583b4090741f681a9dc0da0b80a9"
server_key_iv: "2e90d3677c77f3cda13a84394ac8ea44"
server_key_version: "encryptionkey@0"
I20260809 05:21:42.058604 30690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 05:21:42.062074 30714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:42.063417 30690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:21:42.063592 30690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "589df682b1d24699936a0e75c71ded1f"
format_stamp: "Formatted at 2026-08-09 05:21:42 on dist-test-slave-rmhg"
server_key: "6792583b4090741f681a9dc0da0b80a9"
server_key_iv: "2e90d3677c77f3cda13a84394ac8ea44"
server_key_version: "encryptionkey@0"
I20260809 05:21:42.063791 30690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:42.093293 30690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:42.094166 30690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:42.094444 30690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:42.111649 30690 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:42045
I20260809 05:21:42.111670 30793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:42045 every 8 connection(s)
I20260809 05:21:42.112972 30690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:42.116837 30794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:42.118268 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 30690
I20260809 05:21:42.118458 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:42.119093 22533 external_mini_cluster.cc:1428] Setting key 4db872116aba5e354230b7eaf021aa83
I20260809 05:21:42.130388 30794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f: Bootstrap starting.
I20260809 05:21:42.134362 30794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:42.135087 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:42.121007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.22.1.126:42045)
0809 05:21:42.122196 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:42.122212 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:42.122267 (+ 55us) server_negotiation.cc:407] Connection header received
0809 05:21:42.124521 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:42.124551 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:42.125184 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:42.125622 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:42.126186 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.128254 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.129581 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.130714 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.130895 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:42.132056 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:42.132068 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:42.132088 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:21:42.132834 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:42.132964 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:21:42.132993 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:42.133105 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:42.133404 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 05:21:42.133736 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":353,"threads_started":1}
I20260809 05:21:42.135686 30794 log.cc:824] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:42.138866 30794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f: No bootstrap required, opened a new log
I20260809 05:21:42.139010 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:21:42.146597 30794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } }
I20260809 05:21:42.146991 30794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:42.147094 30794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589df682b1d24699936a0e75c71ded1f, State: Initialized, Role: FOLLOWER
I20260809 05:21:42.147677 30794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } }
I20260809 05:21:42.147903 30794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:42.148041 30794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:42.148208 30794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:42.150496 30794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } }
I20260809 05:21:42.151003 30794 leader_election.cc:302] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 589df682b1d24699936a0e75c71ded1f; no voters:
I20260809 05:21:42.151693 30794 leader_election.cc:288] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:42.151814 30804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:42.152493 30804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [term 1 LEADER]: Becoming Leader. State: Replica: 589df682b1d24699936a0e75c71ded1f, State: Running, Role: LEADER
I20260809 05:21:42.153005 30804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } }
I20260809 05:21:42.153828 30794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:42.155849 30805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "589df682b1d24699936a0e75c71ded1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } } }
I20260809 05:21:42.155957 30806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 589df682b1d24699936a0e75c71ded1f. Latest consensus state: current_term: 1 leader_uuid: "589df682b1d24699936a0e75c71ded1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "589df682b1d24699936a0e75c71ded1f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 42045 } } }
I20260809 05:21:42.156275 30805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:42.156276 30806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:42.160545 30814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:42.164778 30814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:42.173189 30814 catalog_manager.cc:1347] Generated new cluster ID: ef2afa9d6bff48eeb5369c6b7fe5ed63
I20260809 05:21:42.173406 30814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:42.189311 30814 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:42.191071 30814 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:42.205840 30814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 589df682b1d24699936a0e75c71ded1f: Generated new TSK 0
I20260809 05:21:42.206713 30814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:21:42.241847 30799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:42.225900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.22.1.126:42045)
0809 05:21:42.226112 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:42.226119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:42.226693 (+ 574us) server_negotiation.cc:407] Connection header received
0809 05:21:42.226940 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:42.226944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:42.227038 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:42.227168 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0809 05:21:42.227726 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.228493 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.230700 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.231201 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.231307 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:42.231965 (+ 658us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0809 05:21:42.241250 (+ 9285us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 05:21:42.241598 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 05:21:42.242004 30799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260809 05:21:42.242316 30839 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:42.225723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:42045 (local address 127.0.0.1:60630)
0809 05:21:42.226582 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:42.226601 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:42.226758 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:42.227201 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:42.227207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:42.227220 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0809 05:21:42.227547 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:42.227560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.228648 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.228654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:42.230530 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:42.230548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:42.231384 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:42.231392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:42.231629 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:42.231681 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0809 05:21:42.241542 (+ 9861us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0809 05:21:42.241869 (+ 327us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.1.126:42045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":542,"threads_started":1}
I20260809 05:21:42.243324 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 30690
I20260809 05:21:42.244704 30786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:42.345708 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 30690
2026-08-09T05:21:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (743 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (420 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (416 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:43 dist-test-slave-rmhg krb5kdc[30859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:43 dist-test-slave-rmhg krb5kdc[30859](info): set up 2 sockets
Aug 09 05:21:43 dist-test-slave-rmhg krb5kdc[30859](info): commencing operation
krb5kdc: starting...
W20260809 05:21:44.364043 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:44 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:21:44.598521 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36757
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:36757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:44.925207 30879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:44.925529 30879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:44.925661 30879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:44.932940 30879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:44.933058 30879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:44.933111 30879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:44.933218 30879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:44.933281 30879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:44.943306 30879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:36757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:44.944734 30879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:44.946107 30879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:44.958289 30884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:44.958289 30885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:44.958508 30887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:44.959203 30879 server_base.cc:1034] running on GCE node
I20260809 05:21:44.960071 30879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:44.961627 30879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:44.962877 30879 hybrid_clock.cc:648] HybridClock initialized: now 1786252904962813 us; error 43 us; skew 500 ppm
I20260809 05:21:44.963256 30879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:44 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252904, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:44.966486 30879 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:21:44.967962 30879 webserver.cc:460] Webserver started at http://127.22.1.126:35771/ using document root <none> and password file <none>
I20260809 05:21:44.968681 30879 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:44.968791 30879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:44.969111 30879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:44.972230 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "eefed4475a65441fae63cad40a50d8fc"
format_stamp: "Formatted at 2026-08-09 05:21:44 on dist-test-slave-rmhg"
server_key: "6ef8f6842255b92751a446d1a11f4484"
server_key_iv: "a37a9fdca8ebe5ad9ea368ced4ad92e7"
server_key_version: "encryptionkey@0"
I20260809 05:21:44.973053 30879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "eefed4475a65441fae63cad40a50d8fc"
format_stamp: "Formatted at 2026-08-09 05:21:44 on dist-test-slave-rmhg"
server_key: "6ef8f6842255b92751a446d1a11f4484"
server_key_iv: "a37a9fdca8ebe5ad9ea368ced4ad92e7"
server_key_version: "encryptionkey@0"
I20260809 05:21:44.978446 30879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:21:44.982062 30900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:44.983448 30879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:44.983640 30879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "eefed4475a65441fae63cad40a50d8fc"
format_stamp: "Formatted at 2026-08-09 05:21:44 on dist-test-slave-rmhg"
server_key: "6ef8f6842255b92751a446d1a11f4484"
server_key_iv: "a37a9fdca8ebe5ad9ea368ced4ad92e7"
server_key_version: "encryptionkey@0"
I20260809 05:21:44.983822 30879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:45.002686 30879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:45.007129 30879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:45.007406 30879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:45.025189 30879 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:36757
I20260809 05:21:45.025231 30976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:36757 every 8 connection(s)
I20260809 05:21:45.026580 30879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:45.030488 30978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:45.031047 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 30879
I20260809 05:21:45.031320 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:45.032191 22533 external_mini_cluster.cc:1428] Setting key 44d2dcae087f930d7b8e6cfb8b356eae
I20260809 05:21:45.044950 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc: Bootstrap starting.
Aug 09 05:21:45 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:45.048808 30978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:45.049921 30978 log.cc:824] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:45.052656 30978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc: No bootstrap required, opened a new log
I20260809 05:21:45.055241 30982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:45.034638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.22.1.126:36757)
0809 05:21:45.035395 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:45.035408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:45.035457 (+ 49us) server_negotiation.cc:407] Connection header received
0809 05:21:45.037244 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:45.037269 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:45.037700 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:45.038055 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:45.039763 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.041047 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.042811 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.043477 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.043613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:45.048305 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:45.048313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:45.048330 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:45.048372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:45.051338 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:45.051971 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:45.051977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:45.051984 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:45.052151 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:45.052508 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:45.052518 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:45.052520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:45.053062 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:45.053361 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:45.053795 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 05:21:45.054212 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":338,"threads_started":1}
I20260809 05:21:45.060348 30978 raft_consensus.cc:348] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } }
I20260809 05:21:45.060719 30978 raft_consensus.cc:374] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:45.060815 30978 raft_consensus.cc:729] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eefed4475a65441fae63cad40a50d8fc, State: Initialized, Role: FOLLOWER
I20260809 05:21:45.061378 30978 consensus_queue.cc:260] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } }
I20260809 05:21:45.061607 30978 raft_consensus.cc:388] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:45.061731 30978 raft_consensus.cc:482] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:45.061877 30978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:45.064263 30978 raft_consensus.cc:504] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } }
I20260809 05:21:45.064824 30978 leader_election.cc:302] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eefed4475a65441fae63cad40a50d8fc; no voters:
I20260809 05:21:45.065399 30978 leader_election.cc:288] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:45.065513 30986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:45.066180 30986 raft_consensus.cc:686] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [term 1 LEADER]: Becoming Leader. State: Replica: eefed4475a65441fae63cad40a50d8fc, State: Running, Role: LEADER
I20260809 05:21:45.066682 30986 consensus_queue.cc:237] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } }
I20260809 05:21:45.067425 30978 sys_catalog.cc:564] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:45.069458 30988 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eefed4475a65441fae63cad40a50d8fc. Latest consensus state: current_term: 1 leader_uuid: "eefed4475a65441fae63cad40a50d8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } } }
I20260809 05:21:45.069868 30988 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:45.070801 30987 sys_catalog.cc:455] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eefed4475a65441fae63cad40a50d8fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eefed4475a65441fae63cad40a50d8fc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 36757 } } }
I20260809 05:21:45.071063 30987 sys_catalog.cc:458] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:45.071614 30993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:45.077024 30993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:45.086788 30993 catalog_manager.cc:1347] Generated new cluster ID: ea7f1a6d5cdf4e1b8437e56042953526
I20260809 05:21:45.086902 30993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:45.094075 30993 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:45.095355 30993 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:45.105417 30993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eefed4475a65441fae63cad40a50d8fc: Generated new TSK 0
I20260809 05:21:45.106279 30993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:21:45.172138 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:45.524173 31018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:45.524484 31018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:45.524608 31018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:45.531965 31018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:45.532095 31018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:45.532250 31018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:45.542204 31018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:45.543635 31018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:45.545336 31018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:45.558722 31030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:45.558748 31035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:45.558671 31032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:45.559561 31018 server_base.cc:1034] running on GCE node
I20260809 05:21:45.560171 31018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:45.561187 31018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:45.562422 31018 hybrid_clock.cc:648] HybridClock initialized: now 1786252905562367 us; error 44 us; skew 500 ppm
I20260809 05:21:45.562741 31018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:45 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:45.569564 31018 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:21:45.571149 31018 webserver.cc:460] Webserver started at http://127.22.1.65:38795/ using document root <none> and password file <none>
I20260809 05:21:45.571851 31018 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:45.571960 31018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:45.572286 31018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:45.575419 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "436a5822cded4fe1be94e278c4489b14"
format_stamp: "Formatted at 2026-08-09 05:21:45 on dist-test-slave-rmhg"
server_key: "d783d0127727a6a75936cdd3e29efdcb"
server_key_iv: "3c00dd63b7db5661bb3d87537e5a23ab"
server_key_version: "encryptionkey@0"
I20260809 05:21:45.576228 31018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "436a5822cded4fe1be94e278c4489b14"
format_stamp: "Formatted at 2026-08-09 05:21:45 on dist-test-slave-rmhg"
server_key: "d783d0127727a6a75936cdd3e29efdcb"
server_key_iv: "3c00dd63b7db5661bb3d87537e5a23ab"
server_key_version: "encryptionkey@0"
I20260809 05:21:45.581827 31018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 05:21:45.585661 31042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:45.587097 31018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:45.587265 31018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "436a5822cded4fe1be94e278c4489b14"
format_stamp: "Formatted at 2026-08-09 05:21:45 on dist-test-slave-rmhg"
server_key: "d783d0127727a6a75936cdd3e29efdcb"
server_key_iv: "3c00dd63b7db5661bb3d87537e5a23ab"
server_key_version: "encryptionkey@0"
I20260809 05:21:45.587507 31018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:45.625828 31018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:45.630194 31018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:45.630461 31018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:45.631423 31018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:45.632774 31018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:45.632853 31018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:45.632939 31018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:45.632982 31018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:45.663570 31018 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:33895
I20260809 05:21:45.663632 31213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:33895 every 8 connection(s)
I20260809 05:21:45.664909 31018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:45.669473 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 31018
I20260809 05:21:45.669687 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:45.670323 22533 external_mini_cluster.cc:1428] Setting key fda9fa385d0d8c8d731ce7f9c8b4d7e1
Aug 09 05:21:45 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252905, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:45.685014 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:45.667556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56933 (local address 127.22.1.126:36757)
0809 05:21:45.667995 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:45.668001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:45.669047 (+ 1046us) server_negotiation.cc:407] Connection header received
0809 05:21:45.671164 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:45.671168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:45.671237 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:45.671342 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:45.672891 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.673680 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.675181 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.675577 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.675688 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:45.679043 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:45.679049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:45.679052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:45.679086 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:45.682071 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:45.682768 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:45.682773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:45.682776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:45.682920 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:45.683289 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:45.683294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:45.683298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:45.683625 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:45.683754 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:45.684443 (+ 689us) server_negotiation.cc:299] Negotiation successful
0809 05:21:45.684670 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":205,"threads_started":1}
I20260809 05:21:45.685870 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:45.668163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:36757 (local address 127.22.1.65:56933)
0809 05:21:45.668897 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:45.668935 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:45.670876 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:45.671549 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:45.671557 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:45.672147 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:45.672732 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:45.672751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.673832 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.673836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:45.675048 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:45.675060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:45.675792 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:45.675800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:45.676021 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:45.676702 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:45.676725 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:45.678835 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:45.682199 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:45.682206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:45.682218 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:45.682647 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:45.683016 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:45.683019 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:45.683021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:45.683196 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:45.683727 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:45.683733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:45.684062 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 05:21:45.684294 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 05:21:45.684696 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":318,"threads_started":1}
I20260809 05:21:45.687404 31215 heartbeater.cc:347] Connected to a master server at 127.22.1.126:36757
I20260809 05:21:45.687741 31215 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:45.688669 31215 heartbeater.cc:511] Master 127.22.1.126:36757 requested a full tablet report, sending...
I20260809 05:21:45.690874 30931 ts_manager.cc:194] Registered new tserver with Master: 436a5822cded4fe1be94e278c4489b14 (127.22.1.65:33895)
I20260809 05:21:45.692467 30931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:56933
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:21:45.718444 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:46.045179 31226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:46.045521 31226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:46.045661 31226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:46.052968 31226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:46.053104 31226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:46.053285 31226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:46.063232 31226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:46.064599 31226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:46.066008 31226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:46.079192 31231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:46.079138 31233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:46.079164 31235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:46.079936 31226 server_base.cc:1034] running on GCE node
I20260809 05:21:46.080600 31226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:46.081580 31226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:46.082777 31226 hybrid_clock.cc:648] HybridClock initialized: now 1786252906082722 us; error 42 us; skew 500 ppm
I20260809 05:21:46.083089 31226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.086179 31226 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:21:46.087697 31226 webserver.cc:460] Webserver started at http://127.22.1.66:41907/ using document root <none> and password file <none>
I20260809 05:21:46.088366 31226 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:46.088490 31226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:46.088815 31226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:46.091907 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "461459974b3f4ce9bf1e293a05dffa1e"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "1670b8fd05be1f86eddbebb781d1fae9"
server_key_iv: "e0dde0c9348412fc03cedd99a734839a"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.092649 31226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "461459974b3f4ce9bf1e293a05dffa1e"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "1670b8fd05be1f86eddbebb781d1fae9"
server_key_iv: "e0dde0c9348412fc03cedd99a734839a"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.097787 31226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 05:21:46.101348 31246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.102758 31226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:46.102918 31226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "461459974b3f4ce9bf1e293a05dffa1e"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "1670b8fd05be1f86eddbebb781d1fae9"
server_key_iv: "e0dde0c9348412fc03cedd99a734839a"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.103258 31226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:46.118990 31226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:46.123219 31226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:46.123507 31226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:46.124483 31226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:46.125857 31226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:46.125937 31226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.126070 31226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:46.126132 31226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.157182 31226 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:43527
I20260809 05:21:46.157209 31417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:43527 every 8 connection(s)
I20260809 05:21:46.158443 31226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:46.160426 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 31226
I20260809 05:21:46.160636 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:46.161247 22533 external_mini_cluster.cc:1428] Setting key 3c5a92d72f9435acc7f1c19dabfbd0c3
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:46.178826 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.161071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:56589 (local address 127.22.1.126:36757)
0809 05:21:46.161266 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.161271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.162718 (+ 1447us) server_negotiation.cc:407] Connection header received
0809 05:21:46.164587 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.164593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.164675 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.164761 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.166369 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.167165 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.168862 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.169367 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.169491 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.173183 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.173191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.173197 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.173257 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.175819 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.176508 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.176513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.176517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.176657 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.177074 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.177080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.177082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.177398 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.177550 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.178246 (+ 696us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.178484 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:21:46.179852 31422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.161785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:36757 (local address 127.22.1.66:56589)
0809 05:21:46.162574 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:46.162614 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:46.164310 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:46.165020 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:46.165028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:46.165579 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:46.166190 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:46.166208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.167320 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.167324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:46.168726 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:46.168743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.169594 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.169602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:46.169829 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.170537 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:46.170560 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:46.172932 (+ 2372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:46.175955 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:46.175965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:46.175982 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:46.176394 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:46.176763 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:46.176768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:46.176772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:46.176966 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:46.177531 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:46.177539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:46.177892 (+ 353us) client_negotiation.cc:769] Sending connection context
0809 05:21:46.178142 (+ 250us) client_negotiation.cc:240] Negotiation successful
0809 05:21:46.178695 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":303,"threads_started":1}
I20260809 05:21:46.181354 31418 heartbeater.cc:347] Connected to a master server at 127.22.1.126:36757
I20260809 05:21:46.181638 31418 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
I20260809 05:21:46.182375 31418 heartbeater.cc:511] Master 127.22.1.126:36757 requested a full tablet report, sending...
I20260809 05:21:46.183811 30931 ts_manager.cc:194] Registered new tserver with Master: 461459974b3f4ce9bf1e293a05dffa1e (127.22.1.66:43527)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
I20260809 05:21:46.184725 30931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:56589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:21:46.212059 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:46.545873 31426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:46.546180 31426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:46.546303 31426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:46.553526 31426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:46.553646 31426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:46.553794 31426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:46.563581 31426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:36757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:46.564970 31426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:46.566351 31426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:46.580011 31435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:46.580062 31439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:46.580066 31437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:46.580832 31426 server_base.cc:1034] running on GCE node
I20260809 05:21:46.581488 31426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:46.582439 31426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:46.583633 31426 hybrid_clock.cc:648] HybridClock initialized: now 1786252906583570 us; error 59 us; skew 500 ppm
I20260809 05:21:46.583969 31426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.587112 31426 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:21:46.588584 31426 webserver.cc:460] Webserver started at http://127.22.1.67:37143/ using document root <none> and password file <none>
I20260809 05:21:46.589287 31426 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:46.589402 31426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:46.589705 31426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:46.592702 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "384ffcb6e5da4f4bb2757fbdc53208f2"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "44b19f4e3ccad06e8e8005151ea42da5"
server_key_iv: "9fab651bfddef15536611dd902de199d"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.593475 31426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "384ffcb6e5da4f4bb2757fbdc53208f2"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "44b19f4e3ccad06e8e8005151ea42da5"
server_key_iv: "9fab651bfddef15536611dd902de199d"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.598553 31426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260809 05:21:46.602049 31449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.603480 31426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:46.603655 31426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "384ffcb6e5da4f4bb2757fbdc53208f2"
format_stamp: "Formatted at 2026-08-09 05:21:46 on dist-test-slave-rmhg"
server_key: "44b19f4e3ccad06e8e8005151ea42da5"
server_key_iv: "9fab651bfddef15536611dd902de199d"
server_key_version: "encryptionkey@0"
I20260809 05:21:46.603863 31426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:46.618060 31426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:46.622522 31426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:46.622789 31426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:46.623781 31426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:46.625249 31426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:46.625324 31426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.625418 31426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:46.625460 31426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:46.656091 31426 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:37757
I20260809 05:21:46.656136 31616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:37757 every 8 connection(s)
I20260809 05:21:46.657435 31426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:46.665535 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 31426
I20260809 05:21:46.665726 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:46.666280 22533 external_mini_cluster.cc:1428] Setting key 6e9bb56416e0fa44a4aa2f3f348e078f
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:21:46.679286 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.660151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:38681 (local address 127.22.1.126:36757)
0809 05:21:46.660360 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.660366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.661552 (+ 1186us) server_negotiation.cc:407] Connection header received
0809 05:21:46.663584 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.663589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.663679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.663770 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.665965 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.667086 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.669458 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.669860 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.669958 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.673421 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.673426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.673429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.673462 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.676236 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.676949 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.676954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.676958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.677092 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.677543 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.677548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.677552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.677832 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.677961 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.678713 (+ 752us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.678939 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:21:46.680869 31619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.660602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:36757 (local address 127.22.1.67:38681)
0809 05:21:46.661371 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:46.661412 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:46.663233 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:46.664076 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:46.664090 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:46.664892 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:46.665751 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:46.665779 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.667248 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.667254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:46.669312 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:46.669329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.670058 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.670063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:46.670262 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.670922 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:46.670970 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:46.673203 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:46.676369 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:46.676379 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:46.676397 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:46.676822 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:46.677216 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:46.677222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:46.677226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:46.677419 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:46.677959 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:46.677968 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:46.678332 (+ 364us) client_negotiation.cc:769] Sending connection context
0809 05:21:46.678584 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:21:46.679152 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
I20260809 05:21:46.682499 31617 heartbeater.cc:347] Connected to a master server at 127.22.1.126:36757
I20260809 05:21:46.682816 31617 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:46.683584 31617 heartbeater.cc:511] Master 127.22.1.126:36757 requested a full tablet report, sending...
I20260809 05:21:46.685047 30931 ts_manager.cc:194] Registered new tserver with Master: 384ffcb6e5da4f4bb2757fbdc53208f2 (127.22.1.67:37757)
I20260809 05:21:46.685874 30931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:38681
I20260809 05:21:46.694231 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:21:46.695327 31215 heartbeater.cc:503] Master 127.22.1.126:36757 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/test-admin.keytab.
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.722779 22533 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260809 05:21:46.727859 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:46.725491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.22.1.65:33895)
0809 05:21:46.726144 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.726151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.726180 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:21:46.726254 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.726261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.726462 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.726601 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.727470 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54860: BlockingRecv error: recv got EOF from 127.0.0.1:54860 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":349,"threads_started":1}
W20260809 05:21:46.728057 22533 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc' of type FILE
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.729692 22533 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.731022 22533 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:46.745999 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.732657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.22.1.65:33895)
0809 05:21:46.732852 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.732858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.733048 (+ 190us) server_negotiation.cc:407] Connection header received
0809 05:21:46.733269 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.733275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.733338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.733422 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.734365 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.735342 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.736582 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.737229 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.737345 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.740767 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.740778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.740793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.740832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.743370 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.743981 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.743989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.743991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.744114 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.744466 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.744471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.744474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.744818 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.745106 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.745474 (+ 368us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.745730 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 05:21:46.751461 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:46.749911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.22.1.65:33895)
0809 05:21:46.750119 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.750125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.750262 (+ 137us) server_negotiation.cc:407] Connection header received
0809 05:21:46.750391 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.750395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.750447 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.750528 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.751229 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54886: BlockingRecv error: recv got EOF from 127.0.0.1:54886 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260809 05:21:46.752661 22533 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc)
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.754185 22533 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.755409 22533 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:46.769469 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.756858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.22.1.65:33895)
0809 05:21:46.757058 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.757064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.757408 (+ 344us) server_negotiation.cc:407] Connection header received
0809 05:21:46.757611 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.757617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.757703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.757789 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.758765 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.759479 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.760674 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.761100 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.761230 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.764546 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.764551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.764554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.764600 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.766962 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.767657 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.767662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.767664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.767791 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.768144 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.768148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.768150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.768451 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.768704 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.768961 (+ 257us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.769219 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260809 05:21:46.773764 31628 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:21:46.772062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.22.1.65:33895)
0809 05:21:46.772237 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.772243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.772558 (+ 315us) server_negotiation.cc:407] Connection header received
0809 05:21:46.772720 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.772724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.772790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.772866 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.773569 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54896: BlockingRecv error: recv got EOF from 127.0.0.1:54896 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260809 05:21:46.774078 22533 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786252853919692-22533-0/krb5kdc/krb5cc)
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.775556 22533 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:46.776790 22533 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:21:46.790905 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.778062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.22.1.65:33895)
0809 05:21:46.778207 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.778211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.778564 (+ 353us) server_negotiation.cc:407] Connection header received
0809 05:21:46.778724 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.778728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.778801 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.778887 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.779810 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.780532 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.781786 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.782207 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.782308 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.785806 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.785812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.785815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.785848 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.788343 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.788991 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.788995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.788998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.789293 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.789645 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.789649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.789651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.789964 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.790209 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.790410 (+ 201us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.790598 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:21:46.792582 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 31018
I20260809 05:21:46.804034 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.793274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.22.1.65:33895)
0809 05:21:46.793446 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.793453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.793798 (+ 345us) server_negotiation.cc:407] Connection header received
0809 05:21:46.793910 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.793913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.793969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.794039 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.794903 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.795555 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.796648 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.797045 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.797130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.799111 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.799116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.799119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.799149 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.801375 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.801981 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.801984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.801987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.802123 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.802544 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.802548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.802550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.802836 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.803070 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.803504 (+ 434us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.803690 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:21:46.804656 31205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:46.886950 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 31018
I20260809 05:21:46.910746 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 31226
Aug 09 05:21:46 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:21:46.925560 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:46.911658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.22.1.66:43527)
0809 05:21:46.912154 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:46.912161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:46.912187 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:21:46.912251 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:46.912257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:46.912466 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:46.912605 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:46.913536 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.914534 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.915721 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:46.916287 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:46.916389 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:46.920172 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:46.920186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:46.920203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:46.920239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:46.923113 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.923701 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.923707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.923709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.923831 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:46.924113 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:46.924116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:46.924118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:46.924490 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:46.924722 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:46.924953 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 05:21:46.925243 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260809 05:21:46.926847 31409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:47.007936 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 31226
I20260809 05:21:47.029040 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 31426
Aug 09 05:21:47 dist-test-slave-rmhg krb5kdc[30859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:21:47.045096 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:47.030087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50906 (local address 127.22.1.67:37757)
0809 05:21:47.030731 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:47.030738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:47.030768 (+ 30us) server_negotiation.cc:407] Connection header received
0809 05:21:47.030831 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:47.030837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:47.031055 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:47.031180 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:47.032425 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:47.033570 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:47.034748 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:47.035314 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:47.035409 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:47.039347 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:47.039361 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:47.039379 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:47.039418 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:47.042310 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:47.043106 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:47.043114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:47.043116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:47.043260 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:47.043704 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:47.043708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:47.043710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:47.044087 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:47.044347 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:47.044579 (+ 232us) server_negotiation.cc:299] Negotiation successful
0809 05:21:47.044798 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":337,"threads_started":1}
I20260809 05:21:47.046438 31609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:47.127933 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 31426
I20260809 05:21:47.151201 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 30879
I20260809 05:21:47.152396 30971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:47.254245 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 30879
2026-08-09T05:21:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4076 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:47 dist-test-slave-rmhg krb5kdc[31652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:47 dist-test-slave-rmhg krb5kdc[31652](info): set up 2 sockets
Aug 09 05:21:47 dist-test-slave-rmhg krb5kdc[31652](info): commencing operation
krb5kdc: starting...
W20260809 05:21:48.457064 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:48 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:48Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab.
Entry for principal oryx/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab.
I20260809 05:21:48.689076 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41321
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab
--principal=oryx/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:49.022832 31667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:49.023118 31667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:49.023249 31667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:49.030414 31667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:49.030530 31667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:49.030601 31667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:49.030680 31667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:49.030740 31667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:49.040599 31667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab
--principal=oryx/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:49.042043 31667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:49.043455 31667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:49.055094 31673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:49.055152 31674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:49.055188 31677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:49.055750 31667 server_base.cc:1034] running on GCE node
I20260809 05:21:49.056496 31667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:49.058018 31667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:49.059239 31667 hybrid_clock.cc:648] HybridClock initialized: now 1786252909059180 us; error 51 us; skew 500 ppm
I20260809 05:21:49.059585 31667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:49 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252909, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:49.062673 31667 init.cc:377] Logged in from keytab as oryx/127.22.1.126@KRBTEST.COM (short username oryx)
I20260809 05:21:49.064157 31667 webserver.cc:460] Webserver started at http://127.22.1.126:40279/ using document root <none> and password file <none>
I20260809 05:21:49.064843 31667 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:49.064965 31667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:49.065310 31667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:49.068346 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "b5ad151ceb04438cb667b7907a660c9a"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "9e3a124dadae86ef00568fa365382d23"
server_key_iv: "038ab30a3e007f7c18f1c56d055c2bbe"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.069244 31667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "b5ad151ceb04438cb667b7907a660c9a"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "9e3a124dadae86ef00568fa365382d23"
server_key_iv: "038ab30a3e007f7c18f1c56d055c2bbe"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.074460 31667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 05:21:49.078168 31689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:49.079584 31667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:49.079748 31667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "b5ad151ceb04438cb667b7907a660c9a"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "9e3a124dadae86ef00568fa365382d23"
server_key_iv: "038ab30a3e007f7c18f1c56d055c2bbe"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.079933 31667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:49.098137 31667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:49.102566 31667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:49.102844 31667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:49.120169 31667 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41321
I20260809 05:21:49.120206 31763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41321 every 8 connection(s)
I20260809 05:21:49.121650 31667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:49.125502 31764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:49.131855 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 31667
I20260809 05:21:49.132128 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:49.132934 22533 external_mini_cluster.cc:1428] Setting key b41038678784acc52a7ca5894f120709
I20260809 05:21:49.137334 31764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Bootstrap starting.
I20260809 05:21:49.141258 31764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:49.142532 31764 log.cc:824] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:49.145468 31764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: No bootstrap required, opened a new log
Aug 09 05:21:49 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:49.153319 31764 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } }
I20260809 05:21:49.153712 31764 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:49.153810 31764 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ad151ceb04438cb667b7907a660c9a, State: Initialized, Role: FOLLOWER
I20260809 05:21:49.154554 31764 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } }
I20260809 05:21:49.154827 31764 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:49.154973 31764 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:49.155156 31764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:49.156128 31771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:49.135218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.22.1.126:41321)
0809 05:21:49.135996 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:49.136010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:49.136055 (+ 45us) server_negotiation.cc:407] Connection header received
0809 05:21:49.137835 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:49.137860 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:49.138259 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:49.138591 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:49.140490 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.142020 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.143369 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.144071 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.144246 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:49.148359 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:49.148368 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:49.148387 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:49.148439 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:49.151519 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:49.152211 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:49.152218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:49.152225 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:49.152407 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:49.152782 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:49.152786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:49.152788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:49.153388 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:49.153695 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:49.154225 (+ 530us) server_negotiation.cc:299] Negotiation successful
0809 05:21:49.154682 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":346,"threads_started":1}
I20260809 05:21:49.158188 31764 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } }
I20260809 05:21:49.158783 31764 leader_election.cc:302] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ad151ceb04438cb667b7907a660c9a; no voters:
I20260809 05:21:49.159482 31764 leader_election.cc:288] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:49.159582 31776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:49.160245 31776 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 LEADER]: Becoming Leader. State: Replica: b5ad151ceb04438cb667b7907a660c9a, State: Running, Role: LEADER
I20260809 05:21:49.160749 31776 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } }
I20260809 05:21:49.161747 31764 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:49.163602 31779 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ad151ceb04438cb667b7907a660c9a. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } }
I20260809 05:21:49.164014 31779 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:49.163547 31777 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } }
I20260809 05:21:49.164573 31777 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:49.169783 31788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:49.175501 31788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:49.184444 31788 catalog_manager.cc:1347] Generated new cluster ID: 82aabcafccfd4eada520395c3df12c1c
I20260809 05:21:49.184643 31788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:49.210672 31788 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:49.212711 31788 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:49.221062 31788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Generated new TSK 0
I20260809 05:21:49.222211 31788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
I20260809 05:21:49.280906 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:49.620034 31812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:49.620333 31812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:49.620452 31812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:49.627938 31812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:49.628069 31812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:49.628217 31812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:49.638279 31812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:49.639712 31812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:49.641124 31812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:49.654793 31817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:49.654881 31818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:49.655329 31820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:49.655687 31812 server_base.cc:1034] running on GCE node
I20260809 05:21:49.656301 31812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:49.657456 31812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:49.658664 31812 hybrid_clock.cc:648] HybridClock initialized: now 1786252909658616 us; error 47 us; skew 500 ppm
I20260809 05:21:49.658980 31812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:49 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252909, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:49.662176 31812 init.cc:377] Logged in from keytab as oryx/127.22.1.65@KRBTEST.COM (short username oryx)
I20260809 05:21:49.663678 31812 webserver.cc:460] Webserver started at http://127.22.1.65:46311/ using document root <none> and password file <none>
I20260809 05:21:49.664376 31812 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:49.664503 31812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:49.664767 31812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:49.667953 31812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "97539a6b121c4f1aa9cef992d0948676"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "413f2257548f6283df2dc5d16491ed9b"
server_key_iv: "31f7bc9d3fe5fa47c841d7fbfee8d00b"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.668717 31812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "97539a6b121c4f1aa9cef992d0948676"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "413f2257548f6283df2dc5d16491ed9b"
server_key_iv: "31f7bc9d3fe5fa47c841d7fbfee8d00b"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.673938 31812 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 05:21:49.677501 31830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:49.678864 31812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:49.679014 31812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "97539a6b121c4f1aa9cef992d0948676"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "413f2257548f6283df2dc5d16491ed9b"
server_key_iv: "31f7bc9d3fe5fa47c841d7fbfee8d00b"
server_key_version: "encryptionkey@0"
I20260809 05:21:49.679199 31812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:49.713810 31812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:49.717967 31812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:49.718217 31812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:49.719111 31812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:49.720469 31812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:49.720544 31812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:49.720675 31812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:49.720741 31812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:49.750447 31812 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:40229
I20260809 05:21:49.750496 31988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:40229 every 8 connection(s)
I20260809 05:21:49.751765 31812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:49.756738 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 31812
I20260809 05:21:49.756989 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:21:49.757665 22533 external_mini_cluster.cc:1428] Setting key 6b15087d7ea548a9f507effb4ebbc7b1
Aug 09 05:21:49 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252909, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:49.772607 31997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:49.754296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:50145 (local address 127.22.1.126:41321)
0809 05:21:49.754787 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:49.754792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:49.755739 (+ 947us) server_negotiation.cc:407] Connection header received
0809 05:21:49.757709 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:49.757716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:49.757800 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:49.757897 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:49.759872 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.760613 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.762400 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.762886 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.763010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:49.766405 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:49.766410 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:49.766414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:49.766446 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:49.769251 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:49.770165 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:49.770171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:49.770175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:49.770307 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:49.770752 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:49.770756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:49.770758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:49.771103 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:49.771235 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:49.771999 (+ 764us) server_negotiation.cc:299] Negotiation successful
0809 05:21:49.772237 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":274,"threads_started":1}
I20260809 05:21:49.773671 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:49.754791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:50145)
0809 05:21:49.755552 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:49.755602 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:49.757424 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:49.758165 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:49.758176 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:49.758882 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:49.759701 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:49.759727 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.760768 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.760773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:49.762254 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:49.762267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:49.763123 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:49.763131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:49.763374 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:49.764013 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:49.764042 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:49.766185 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:49.769371 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:49.769380 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:49.769398 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:49.770037 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:49.770415 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:49.770420 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:49.770424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:49.770632 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:49.771224 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:49.771232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:49.771588 (+ 356us) client_negotiation.cc:769] Sending connection context
0809 05:21:49.771850 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 05:21:49.772411 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":319,"threads_started":1}
I20260809 05:21:49.775528 31991 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:49.775851 31991 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:49.776678 31991 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:49.778950 31715 ts_manager.cc:194] Registered new tserver with Master: 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:49.780457 31715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:50145
WARNING: no policy specified for oryx/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
I20260809 05:21:49.814565 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:50.160888 32001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:50.161240 32001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:50.161382 32001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:50.168936 32001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:50.169093 32001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:50.169284 32001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:50.179504 32001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:50.180887 32001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:50.182281 32001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:50.195410 32006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:50.195497 32009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:50.195636 32012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:50.196188 32001 server_base.cc:1034] running on GCE node
I20260809 05:21:50.196753 32001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:50.197687 32001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:50.198875 32001 hybrid_clock.cc:648] HybridClock initialized: now 1786252910198823 us; error 42 us; skew 500 ppm
I20260809 05:21:50.199179 32001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:50 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252910, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:50.202246 32001 init.cc:377] Logged in from keytab as oryx/127.22.1.66@KRBTEST.COM (short username oryx)
I20260809 05:21:50.203729 32001 webserver.cc:460] Webserver started at http://127.22.1.66:46271/ using document root <none> and password file <none>
I20260809 05:21:50.204407 32001 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:50.204541 32001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:50.204829 32001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:50.207762 32001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "2e4efbab91644fa48b2e95d8054b790e"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "0d1e9d8061d586e0777f564cdc208a21"
server_key_iv: "1800046f2375481b4c84dd0a61bc67c4"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.208508 32001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "2e4efbab91644fa48b2e95d8054b790e"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "0d1e9d8061d586e0777f564cdc208a21"
server_key_iv: "1800046f2375481b4c84dd0a61bc67c4"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.213485 32001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260809 05:21:50.216840 32021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.218241 32001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:50.218402 32001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "2e4efbab91644fa48b2e95d8054b790e"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "0d1e9d8061d586e0777f564cdc208a21"
server_key_iv: "1800046f2375481b4c84dd0a61bc67c4"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.218595 32001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:50.235242 32001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:50.239439 32001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:50.239727 32001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:50.240739 32001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:50.242195 32001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:50.242276 32001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.242403 32001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:50.242456 32001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.273756 32001 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:43763
I20260809 05:21:50.273805 32197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:43763 every 8 connection(s)
I20260809 05:21:50.275185 32001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:50.280607 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 32001
I20260809 05:21:50.280814 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:21:50.281370 22533 external_mini_cluster.cc:1428] Setting key 2734b7aa4bffacca5d557c66f60aa00b
Aug 09 05:21:50 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252910, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:50.295485 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:50.277715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:56425 (local address 127.22.1.126:41321)
0809 05:21:50.278214 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:50.278221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:50.279244 (+ 1023us) server_negotiation.cc:407] Connection header received
0809 05:21:50.281065 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:50.281072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:50.281188 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:50.281307 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:50.283027 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.283962 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.285657 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.286100 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.286215 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:50.289844 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:50.289850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:50.289854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:50.289887 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:50.292428 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:50.293137 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:50.293143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:50.293147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:50.293324 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:50.293744 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:50.293750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:50.293753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:50.294039 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:50.294167 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:50.294858 (+ 691us) server_negotiation.cc:299] Negotiation successful
0809 05:21:50.295101 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260809 05:21:50.296518 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:50.278322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:56425)
0809 05:21:50.279067 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:50.279109 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:50.280724 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:50.281536 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:50.281545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:50.282164 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:50.282851 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:50.282873 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.284119 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.284125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:50.285519 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:50.285532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.286323 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.286330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:50.286574 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:50.287219 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:50.287241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:50.289605 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:50.292574 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:50.292584 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:50.292603 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:50.293020 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:50.293427 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:50.293432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:50.293436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:50.293638 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:50.294160 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:50.294167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:50.294503 (+ 336us) client_negotiation.cc:769] Sending connection context
0809 05:21:50.294766 (+ 263us) client_negotiation.cc:240] Negotiation successful
0809 05:21:50.295317 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":287,"threads_started":1}
I20260809 05:21:50.298099 32198 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:50.298466 32198 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:50.299399 32198 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:50.300985 31715 ts_manager.cc:194] Registered new tserver with Master: 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:50.301832 31715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:56425
WARNING: no policy specified for oryx/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
I20260809 05:21:50.336424 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:50.678380 32205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:50.678714 32205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:50.678860 32205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:50.686373 32205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:50.686497 32205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:50.686658 32205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:50.696559 32205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:50.697978 32205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:50.699369 32205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:50.712612 32217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:50.712723 32221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:50.712637 32215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:50.713209 32205 server_base.cc:1034] running on GCE node
I20260809 05:21:50.713819 32205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:50.714680 32205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:50.715874 32205 hybrid_clock.cc:648] HybridClock initialized: now 1786252910715826 us; error 35 us; skew 500 ppm
I20260809 05:21:50.716178 32205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:50 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252910, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:50.722615 32205 init.cc:377] Logged in from keytab as oryx/127.22.1.67@KRBTEST.COM (short username oryx)
I20260809 05:21:50.724079 32205 webserver.cc:460] Webserver started at http://127.22.1.67:35841/ using document root <none> and password file <none>
I20260809 05:21:50.724797 32205 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:50.724918 32205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:50.725270 32205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:50.728259 32205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "d08f2e0339164b1a8d51add7961ad3a5"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "bdb596c2cfcc431b8e2eb09e57382247"
server_key_iv: "add27c84177b733a5ddb036475ca8fdc"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.729215 32205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "d08f2e0339164b1a8d51add7961ad3a5"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "bdb596c2cfcc431b8e2eb09e57382247"
server_key_iv: "add27c84177b733a5ddb036475ca8fdc"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.734611 32205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:21:50.738147 32229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.739512 32205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:50.739662 32205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "d08f2e0339164b1a8d51add7961ad3a5"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "bdb596c2cfcc431b8e2eb09e57382247"
server_key_iv: "add27c84177b733a5ddb036475ca8fdc"
server_key_version: "encryptionkey@0"
I20260809 05:21:50.739848 32205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:50.774499 32205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:50.779081 32205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:50.779362 32205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:50.780345 32205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:50.781749 32205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:50.781831 32205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.781958 32205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:50.782020 32205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:50.783650 31991 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
I20260809 05:21:50.812515 32205 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40243
I20260809 05:21:50.812597 32398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40243 every 8 connection(s)
I20260809 05:21:50.813838 32205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:50.820220 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 32205
I20260809 05:21:50.820405 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:21:50.820830 22533 external_mini_cluster.cc:1428] Setting key 979fbce8e5e66931a4049ab47d12086d
Aug 09 05:21:50 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252910, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:50.834007 32401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:50.816438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:51811 (local address 127.22.1.126:41321)
0809 05:21:50.816893 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:50.816897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:50.817847 (+ 950us) server_negotiation.cc:407] Connection header received
0809 05:21:50.819649 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:50.819655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:50.819743 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:50.819841 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:50.821573 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.822515 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.824204 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.824652 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.824766 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:50.828289 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:50.828294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:50.828297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:50.828331 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:50.830890 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:50.831627 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:50.831631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:50.831633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:50.831757 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:50.832189 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:50.832192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:50.832194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:50.832480 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:50.832621 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:50.833395 (+ 774us) server_negotiation.cc:299] Negotiation successful
0809 05:21:50.833635 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":211,"threads_started":1}
I20260809 05:21:50.835043 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:50.816981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:51811)
0809 05:21:50.817690 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:50.817730 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:50.819364 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:50.820110 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:50.820119 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:50.820713 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:50.821388 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:50.821410 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.822669 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.822673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:50.824060 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:50.824074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:50.824887 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:50.824895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:50.825140 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:50.825830 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:50.825861 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:50.828070 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:50.831036 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:50.831046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:50.831065 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:50.831497 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:50.831864 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:50.831870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:50.831874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:50.832073 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:50.832606 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:50.832614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:50.832988 (+ 374us) client_negotiation.cc:769] Sending connection context
0809 05:21:50.833283 (+ 295us) client_negotiation.cc:240] Negotiation successful
0809 05:21:50.833843 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260809 05:21:50.836612 32399 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:50.836905 32399 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:50.837671 32399 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:50.839133 31715 ts_manager.cc:194] Registered new tserver with Master: d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243)
I20260809 05:21:50.839926 31715 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:51811
I20260809 05:21:50.848941 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.1.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.125.keytab.
Entry for principal oryx/127.22.1.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.125.keytab.
I20260809 05:21:50.902199 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42903
--webserver_interface=127.22.1.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.125.keytab
--principal=oryx/127.22.1.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41321,127.22.1.125:42903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:51.250226 32406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:51.250528 32406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:51.250625 32406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:51.257899 32406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:51.258010 32406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:51.258064 32406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:51.258119 32406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:51.258162 32406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:51.268529 32406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41321,127.22.1.125:42903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.125.keytab
--principal=oryx/127.22.1.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.1.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
W20260809 05:21:51.268957 32406 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.1.126:41321,127.22.1.125:42903'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260809 05:21:51.270334 32406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:51.271808 32406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:51.284552 32413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:51.284468 32418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:51.284651 32415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:51.285712 32406 server_base.cc:1034] running on GCE node
I20260809 05:21:51.286482 32406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:51.287909 32406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:51.289139 32406 hybrid_clock.cc:648] HybridClock initialized: now 1786252911289058 us; error 58 us; skew 500 ppm
I20260809 05:21:51.289503 32406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252911, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:51.292675 32406 init.cc:377] Logged in from keytab as oryx/127.22.1.125@KRBTEST.COM (short username oryx)
I20260809 05:21:51.294253 32406 webserver.cc:460] Webserver started at http://127.22.1.125:44123/ using document root <none> and password file <none>
I20260809 05:21:51.294948 32406 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:51.295039 32406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:51.295307 32406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:51.298378 32406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/instance:
uuid: "19299c1b7af24eb4af96c113298b06e5"
format_stamp: "Formatted at 2026-08-09 05:21:51 on dist-test-slave-rmhg"
server_key: "0466bb6ba0ce710034b0fb5857717921"
server_key_iv: "83f8d1f9ab52e1afcf447d84b8ab785a"
server_key_version: "encryptionkey@0"
I20260809 05:21:51.299139 32406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal/instance:
uuid: "19299c1b7af24eb4af96c113298b06e5"
format_stamp: "Formatted at 2026-08-09 05:21:51 on dist-test-slave-rmhg"
server_key: "0466bb6ba0ce710034b0fb5857717921"
server_key_iv: "83f8d1f9ab52e1afcf447d84b8ab785a"
server_key_version: "encryptionkey@0"
I20260809 05:21:51.304596 32406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:21:51.305132 32198 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
I20260809 05:21:51.308779 32428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:51.310277 32406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 05:21:51.310449 32406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "19299c1b7af24eb4af96c113298b06e5"
format_stamp: "Formatted at 2026-08-09 05:21:51 on dist-test-slave-rmhg"
server_key: "0466bb6ba0ce710034b0fb5857717921"
server_key_iv: "83f8d1f9ab52e1afcf447d84b8ab785a"
server_key_version: "encryptionkey@0"
I20260809 05:21:51.310662 32406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:51.335702 32406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:51.339877 32406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:51.340149 32406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:51.357986 32406 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.125:42903
I20260809 05:21:51.358006 32505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.125:42903 every 8 connection(s)
I20260809 05:21:51.359308 32406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
I20260809 05:21:51.363144 32506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:51.367255 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 32406
I20260809 05:21:51.370683 32506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252911, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.125@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:51.390524 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.374565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.22.1.126:41321)
0809 05:21:51.375041 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.375046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.375550 (+ 504us) server_negotiation.cc:407] Connection header received
0809 05:21:51.375749 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.375754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.375849 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.375968 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.377102 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.377871 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.379398 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.379820 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.379911 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.383758 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.383765 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.383770 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.383814 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.387197 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.387981 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.387985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.387987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.388107 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.388551 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.388556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.388559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.388944 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.389104 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.389961 (+ 857us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.390199 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
I20260809 05:21:51.391530 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.374484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43374)
0809 05:21:51.375387 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.375440 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.375626 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.376024 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.376029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.376669 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.376950 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.376959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.378012 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.378017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.379262 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.379276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.380046 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.380055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.380236 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.380822 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.380853 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.383514 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.387323 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.387333 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.387343 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.387862 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.388209 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.388213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.388216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.388429 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.389077 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.389084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.389543 (+ 459us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.389873 (+ 330us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.390395 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":30,"spinlock_wait_cycles":15744,"thread_start_us":535,"threads_started":1}
I20260809 05:21:51.391546 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.369724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38824 (local address 127.22.1.125:42903)
0809 05:21:51.370530 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.370542 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.370586 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:21:51.372648 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.372675 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.373135 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.373584 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.375277 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.376902 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.378292 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.378954 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.379099 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.384255 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.384267 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.384285 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.384321 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.387593 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.388330 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.388337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.388339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.388473 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.388847 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.388851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.388854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.389498 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.389718 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.390130 (+ 412us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.390699 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":319,"server-negotiator.queue_time_us":588,"thread_start_us":339,"threads_started":1}
I20260809 05:21:51.394165 32506 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252911, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.125@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:51.408043 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.394935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38830)
0809 05:21:51.395201 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.395221 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.395323 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.396478 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.396483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.396880 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.397024 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.397032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.398356 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.398362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.399303 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.399316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.400151 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.400155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.400266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.400933 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.400951 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.403359 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.406016 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.406020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.406023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.406385 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.406698 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.406700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.406703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.406833 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.407300 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.407303 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.407425 (+ 122us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.407531 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.407750 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260809 05:21:51.408114 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.395126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38830 (local address 127.22.1.125:42903)
0809 05:21:51.396196 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.396201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.396219 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:21:51.396267 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.396272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.396340 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.396435 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.397180 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.398222 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.399433 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.399893 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.399996 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.403547 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.403552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.403556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.403585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.405878 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.406480 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.406484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.406486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.406610 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.406916 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.406920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.406922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.407211 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.407315 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.407594 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.407863 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":954}
I20260809 05:21:51.412928 32506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Bootstrap starting.
I20260809 05:21:51.416293 32506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:51.417418 32506 log.cc:824] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:51.420163 32506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: No bootstrap required, opened a new log
I20260809 05:21:51.427191 32506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } }
I20260809 05:21:51.427527 32506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:51.427604 32506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19299c1b7af24eb4af96c113298b06e5, State: Initialized, Role: FOLLOWER
I20260809 05:21:51.428170 32506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } }
I20260809 05:21:51.429020 32516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } }
I20260809 05:21:51.429401 32516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:51.430034 32506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 05:21:51.438665 32529 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:21:51.438802 32529 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:21:51.441634 32406 master_runner.cc:428] Detected that this master 19299c1b7af24eb4af96c113298b06e5 is joining an existing cluster
I20260809 05:21:51.441727 32406 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.1.125:42903
I20260809 05:21:51.462916 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.446248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.22.1.125:42903)
0809 05:21:51.446423 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.446428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.447834 (+ 1406us) server_negotiation.cc:407] Connection header received
0809 05:21:51.448034 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.448038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.448107 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.448176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.449673 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.450698 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.452709 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.453319 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.453408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.456001 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.456008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.456013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.456051 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.459608 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.460534 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.460539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.460542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.460699 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.461118 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.461122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.461125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.461570 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.461709 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.462292 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.462534 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54,"spinlock_wait_cycles":47872}
I20260809 05:21:51.463078 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.445485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.22.1.126:41321)
0809 05:21:51.446002 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.446008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.446503 (+ 495us) server_negotiation.cc:407] Connection header received
0809 05:21:51.446794 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.446800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.446905 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.446991 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.449270 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.450388 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.452657 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.453384 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.453501 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.456342 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.456349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.456355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.456398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.459772 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.460688 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.460693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.460697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.460853 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.461366 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.461371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.461375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.461730 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.461863 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.462485 (+ 622us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.462726 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359}
I20260809 05:21:51.464139 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.445604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43386)
0809 05:21:51.446396 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.446414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.446634 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.447787 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.447792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.448591 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.449041 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.449059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.450575 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.450581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.452449 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.452465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.454216 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.454223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.454383 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.454920 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.454948 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.456108 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.459948 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.459955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.459959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.460539 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.461003 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.461008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.461012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.461228 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.461962 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.461966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.462180 (+ 214us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.463342 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.463666 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":28,"thread_start_us":1463,"threads_started":1}
I20260809 05:21:51.465277 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.445646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38840)
0809 05:21:51.447745 (+ 2099us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.447762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.447923 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.448360 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.448364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.449213 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.449536 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.449547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.450978 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.450983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.452578 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.452592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.453725 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.453735 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.453888 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.454679 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.454700 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.455794 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.459746 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.459750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.459753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.460395 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.460832 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.460841 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.460845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.461000 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.461795 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.461799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.462015 (+ 216us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.463644 (+ 1629us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.463953 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1976,"mutex_wait_us":4,"thread_start_us":1777,"threads_started":1}
WARNING: no policy specified for oryx/127.22.1.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.124@KRBTEST.COM" created.
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252909, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.126@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:51.490156 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.472500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.22.1.125:42903)
0809 05:21:51.472677 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.472683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.473328 (+ 645us) server_negotiation.cc:407] Connection header received
0809 05:21:51.473738 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.473745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.473859 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.473969 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.475438 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.476570 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.478505 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.479159 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.479278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.483455 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.483470 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.483475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.483512 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.486958 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.487911 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.487917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.487920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.488084 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.488526 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.488530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.488533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.488874 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.489007 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.489643 (+ 636us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.489874 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:21:51.490327 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.472110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38850)
0809 05:21:51.473191 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.473241 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.473504 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.474036 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.474044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.474944 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.475257 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.475271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.476732 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.476738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.478355 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.478369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.479376 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.479385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.479581 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.480328 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.480346 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.483215 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.487129 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.487145 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.487154 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.487755 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.488204 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.488209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.488213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.488401 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.489040 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.489049 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.489298 (+ 249us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.489519 (+ 221us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.489884 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":633,"threads_started":1}
I20260809 05:21:51.492553 31715 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.1.125:42903: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "19299c1b7af24eb4af96c113298b06e5"
member_type: NON_VOTER
last_known_addr {
host: "127.22.1.125"
port: 42903
}
attrs {
promote: true
}
}
dest_uuid: "b5ad151ceb04438cb667b7907a660c9a"
cas_config_opid_index: -1
I20260809 05:21:51.494258 31715 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } }
I20260809 05:21:51.506064 32463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:51.505676 32547 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } }
I20260809 05:21:51.509610 32463 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260809 05:21:51.510964 32547 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.1.125:42903
I20260809 05:21:51.511210 32549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:51.511875 32406 master.cc:537] Master@127.22.1.125:42903 shutting down...
I20260809 05:21:51.512034 32546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } }
I20260809 05:21:51.513376 32546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.124.keytab.
Entry for principal oryx/127.22.1.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.124.keytab.
W20260809 05:21:51.516783 31692 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260809 05:21:51.517822 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:42515
--webserver_interface=127.22.1.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.124.keytab
--principal=oryx/127.22.1.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:51.519762 31692 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a -> Peer 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125:42903): Couldn't send request to peer 19299c1b7af24eb4af96c113298b06e5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260809 05:21:51.528024 32406 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:51.528470 32406 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:51.528676 32406 tablet_replica.cc:330] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: stopping tablet replica
I20260809 05:21:51.535037 32406 master.cc:552] Master@127.22.1.125:42903 shutdown complete.
I20260809 05:21:51.535171 32406 master_runner.cc:305] Clearing existing system tablet
I20260809 05:21:51.535590 32406 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:51.535686 32406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:51.537928 32406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260809 05:21:51.539310 32557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:51.539608 32406 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:51.539870 32406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260809 05:21:51.539963 32406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "19299c1b7af24eb4af96c113298b06e5"
format_stamp: "Formatted at 2026-08-09 05:21:51 on dist-test-slave-rmhg"
server_key: "0466bb6ba0ce710034b0fb5857717921"
server_key_iv: "83f8d1f9ab52e1afcf447d84b8ab785a"
server_key_version: "encryptionkey@0"
I20260809 05:21:51.544277 32406 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:51.549122 32406 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 05:21:51.549363 32406 log.cc:1196] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260809 05:21:51.549757 32406 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Deleting consensus metadata
I20260809 05:21:51.549988 32406 master_runner.cc:315] Copying system tablet from 127.22.1.126:41321
I20260809 05:21:51.552664 32406 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: tablet copy: Beginning tablet copy session from remote peer at address 127.22.1.126:41321
I20260809 05:21:51.565606 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.553913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.22.1.126:41321)
0809 05:21:51.554115 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:51.554122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:51.554205 (+ 83us) server_negotiation.cc:407] Connection header received
0809 05:21:51.554475 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:51.554482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:51.554552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:51.554654 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:51.555471 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.556260 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.557770 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.558227 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.558347 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.560130 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:51.560136 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:51.560139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:51.560173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:51.562888 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.563576 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.563581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.563583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.563717 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:51.564110 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:51.564114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:51.564116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:51.564401 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:51.564560 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:51.564948 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 05:21:51.565201 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 05:21:51.566056 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:51.553623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43396)
0809 05:21:51.554092 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:51.554117 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:51.554320 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:51.554680 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:51.554685 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:51.555107 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:51.555318 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.555327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.556529 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.556534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.557609 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:51.557624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:51.558386 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:51.558392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:51.558578 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:51.559189 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:51.559208 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:51.559924 (+ 716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:51.563027 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.563032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.563037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.563456 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.563829 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:51.563835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:51.563839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:51.563999 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:51.564540 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:51.564545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:51.564707 (+ 162us) client_negotiation.cc:769] Sending connection context
0809 05:21:51.564830 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 05:21:51.565201 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":504,"threads_started":1}
I20260809 05:21:51.567008 31741 tablet_copy_service.cc:140] P b5ad151ceb04438cb667b7907a660c9a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 19299c1b7af24eb4af96c113298b06e5 ({username='oryx', principal='oryx/127.22.1.125@KRBTEST.COM'} at 127.0.0.1:43396)
I20260809 05:21:51.567412 31741 tablet_copy_service.cc:161] P b5ad151ceb04438cb667b7907a660c9a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 19299c1b7af24eb4af96c113298b06e5 at {username='oryx', principal='oryx/127.22.1.125@KRBTEST.COM'} at 127.0.0.1:43396: session id = 19299c1b7af24eb4af96c113298b06e5-00000000000000000000000000000000
I20260809 05:21:51.571074 31741 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 05:21:51.573657 32406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 05:21:51.578612 32406 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: tablet copy: Starting download of 0 data blocks...
I20260809 05:21:51.578996 32406 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: tablet copy: Starting download of 1 WAL segments...
I20260809 05:21:51.582532 32406 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 05:21:51.586635 31741 tablet_copy_service.cc:342] P b5ad151ceb04438cb667b7907a660c9a: Request end of tablet copy session 19299c1b7af24eb4af96c113298b06e5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.1.125@KRBTEST.COM'} at 127.0.0.1:43396
I20260809 05:21:51.586882 31741 tablet_copy_service.cc:433] P b5ad151ceb04438cb667b7907a660c9a: ending tablet copy session 19299c1b7af24eb4af96c113298b06e5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 19299c1b7af24eb4af96c113298b06e5
I20260809 05:21:51.599787 32406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:51.605607 32569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:51.607143 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:51.614238 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:51.614837 32406 server_base.cc:1034] running on GCE node
I20260809 05:21:51.615216 32406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:51.615916 32406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:51.617206 32406 hybrid_clock.cc:648] HybridClock initialized: now 1786252911617089 us; error 38 us; skew 500 ppm
I20260809 05:21:51.617439 32406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:51 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252911, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:51.619697 32406 init.cc:377] Logged in from keytab as oryx/127.22.1.125@KRBTEST.COM (short username oryx)
I20260809 05:21:51.621114 32406 webserver.cc:460] Webserver started at http://127.22.1.125:39485/ using document root <none> and password file <none>
I20260809 05:21:51.621560 32406 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:51.621641 32406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:51.624994 32406 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260809 05:21:51.627360 32587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:51.627913 32406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260809 05:21:51.628077 32406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "19299c1b7af24eb4af96c113298b06e5"
format_stamp: "Formatted at 2026-08-09 05:21:51 on dist-test-slave-rmhg"
server_key: "0466bb6ba0ce710034b0fb5857717921"
server_key_iv: "83f8d1f9ab52e1afcf447d84b8ab785a"
server_key_version: "encryptionkey@0"
I20260809 05:21:51.628258 32406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:51.644781 32406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:51.649130 32406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:51.674203 32406 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.125:42903
I20260809 05:21:51.674244 32662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.125:42903 every 8 connection(s)
I20260809 05:21:51.675217 32406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
I20260809 05:21:51.680318 32664 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:21:51.684231 32664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Bootstrap starting.
I20260809 05:21:51.704051 32664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:21:51.704737 32664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: Bootstrap complete.
I20260809 05:21:51.706034 32664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } }
I20260809 05:21:51.706462 32664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19299c1b7af24eb4af96c113298b06e5, State: Initialized, Role: LEARNER
I20260809 05:21:51.706930 32664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } }
I20260809 05:21:51.708187 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } }
I20260809 05:21:51.708603 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: LEARNER
I20260809 05:21:51.708778 32664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:51.727703 32687 catalog_manager.cc:1259] Loaded cluster ID: 82aabcafccfd4eada520395c3df12c1c
I20260809 05:21:51.727825 32687 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: loading cluster ID for follower catalog manager: success
I20260809 05:21:51.731910 32687 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: acquiring CA information for follower catalog manager: success
I20260809 05:21:51.735266 32687 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:21:51.842648 32399 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
W20260809 05:21:52.011670 32553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:52.011981 32553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:52.012077 32553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:52.019362 32553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:52.019484 32553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:52.019531 32553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:52.019587 32553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:52.019631 32553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:52.030026 32553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.124.keytab
--principal=oryx/127.22.1.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:42515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.1.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:52.031430 32553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:52.032756 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:52.045655 32696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.045749 32697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.045900 32699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:52.046980 32553 server_base.cc:1034] running on GCE node
I20260809 05:21:52.047716 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:52.049180 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:52.050395 32553 hybrid_clock.cc:648] HybridClock initialized: now 1786252912050331 us; error 55 us; skew 500 ppm
I20260809 05:21:52.050740 32553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252912, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:52.053804 32553 init.cc:377] Logged in from keytab as oryx/127.22.1.124@KRBTEST.COM (short username oryx)
I20260809 05:21:52.055361 32553 webserver.cc:460] Webserver started at http://127.22.1.124:43159/ using document root <none> and password file <none>
I20260809 05:21:52.056047 32553 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:52.056154 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:52.056452 32553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:52.059461 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/instance:
uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
format_stamp: "Formatted at 2026-08-09 05:21:52 on dist-test-slave-rmhg"
server_key: "56271f1010d0315db20b2622cbfa4e4a"
server_key_iv: "549a8f0ab65710938536c560fdf9a9e1"
server_key_version: "encryptionkey@0"
I20260809 05:21:52.060233 32553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal/instance:
uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
format_stamp: "Formatted at 2026-08-09 05:21:52 on dist-test-slave-rmhg"
server_key: "56271f1010d0315db20b2622cbfa4e4a"
server_key_iv: "549a8f0ab65710938536c560fdf9a9e1"
server_key_version: "encryptionkey@0"
I20260809 05:21:52.065696 32553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 05:21:52.069038 32709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:52.070400 32553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:21:52.070575 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
format_stamp: "Formatted at 2026-08-09 05:21:52 on dist-test-slave-rmhg"
server_key: "56271f1010d0315db20b2622cbfa4e4a"
server_key_iv: "549a8f0ab65710938536c560fdf9a9e1"
server_key_version: "encryptionkey@0"
I20260809 05:21:52.070775 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:52.086941 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:52.090832 32553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:52.091087 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:52.137444 32553 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.124:42515
I20260809 05:21:52.137629 322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.124:42515 every 8 connection(s)
I20260809 05:21:52.139348 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
I20260809 05:21:52.139421 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.121679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.22.1.125:42903)
0809 05:21:52.129567 (+ 7888us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.129573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.129596 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:21:52.129679 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.129684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.129795 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.130773 (+ 978us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:52.131083 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.133062 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.136488 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.137988 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.138155 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.138397 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.138785 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.138932 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7644,"thread_start_us":7484,"threads_started":1}
I20260809 05:21:52.139638 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.121170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38858)
0809 05:21:52.122545 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.122566 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.122889 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.130175 (+ 7286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.130181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.130221 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:52.130593 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.130612 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.133315 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.133322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.136243 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.136262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.138252 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.138260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.138561 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.138627 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.138733 (+ 106us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.138892 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1233,"thread_start_us":1901,"threads_started":1}
I20260809 05:21:52.141528 32629 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260809 05:21:52.142448 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ad151ceb04438cb667b7907a660c9a. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } }
I20260809 05:21:52.142802 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: LEARNER
I20260809 05:21:52.143483 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 32553
I20260809 05:21:52.148113 323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:52.160862 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252912, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.124@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:52.178479 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.145945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.22.1.124:42515)
0809 05:21:52.147389 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.147402 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.152909 (+ 5507us) server_negotiation.cc:407] Connection header received
0809 05:21:52.155697 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.155727 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.156248 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.156761 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.158701 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.160400 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.161786 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.162642 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.162841 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.168844 (+ 6003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.168856 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.168871 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.168917 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.173553 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.174475 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.174482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.174489 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.174668 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.175255 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.175260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.175263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.176114 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.176433 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.176955 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.177413 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":871,"threads_started":1}
I20260809 05:21:52.184039 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.163957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43406)
0809 05:21:52.165498 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.165549 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.165879 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.166333 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.166341 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.167395 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.167836 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.167847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.169481 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.169488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.171140 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.171154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.172570 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.172576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.172744 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.173395 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.173416 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.176857 (+ 3441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.181344 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.181352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.181357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.181872 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.182244 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.182247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.182250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.182375 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.183000 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.183006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.183206 (+ 200us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.183371 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.183652 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":894,"threads_started":1}
I20260809 05:21:52.184079 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.163885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.22.1.126:41321)
0809 05:21:52.164571 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.164578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.165796 (+ 1218us) server_negotiation.cc:407] Connection header received
0809 05:21:52.165991 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.165995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.166076 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.166248 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.167999 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.169319 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.171292 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.172358 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.172486 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.177181 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.177189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.177195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.177235 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.181226 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.181987 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.181991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.181993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.182158 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.182449 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.182451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.182453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.182845 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.182998 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.183522 (+ 524us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.183718 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":392,"threads_started":1}
I20260809 05:21:52.185400 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252912, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.124@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:52.200177 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.186150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.22.1.125:42903)
0809 05:21:52.186314 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.186318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.186459 (+ 141us) server_negotiation.cc:407] Connection header received
0809 05:21:52.186581 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.186585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.186644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.186720 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.187416 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.188280 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.190128 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.190575 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.190675 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.194660 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.194666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.194669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.194702 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.197565 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.198178 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.198183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.198185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.198375 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.198744 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.198748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.198750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.199062 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.199202 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.199540 (+ 338us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.199789 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:21:52.200232 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.186216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38874)
0809 05:21:52.186396 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.186415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.186504 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.186744 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.186747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.187126 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.187258 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.187266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.188468 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.188472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.189989 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.190003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.190749 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.190755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.191043 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.191710 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.191727 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.194391 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.197691 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.197697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.197702 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.198067 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.198489 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.198495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.198499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.198643 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.199179 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.199184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.199321 (+ 137us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.199437 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.199793 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 05:21:52.201442 323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252912, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.124@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
I20260809 05:21:52.217093 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.202033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46090)
0809 05:21:52.202313 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.202331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.202439 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.202702 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.202707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.203231 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.203489 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.203499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.204563 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.204569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.205849 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.205861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.206757 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.206765 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.207157 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.207792 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.207808 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.210442 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.214313 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.214317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.214320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.214887 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.215312 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.215316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.215318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.215548 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.216224 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.216228 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.216378 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.216511 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.216698 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 05:21:52.217398 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.202138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.22.1.124:42515)
0809 05:21:52.202288 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.202293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.202391 (+ 98us) server_negotiation.cc:407] Connection header received
0809 05:21:52.202543 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.202547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.202599 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.202672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.203617 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.204426 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.205981 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.206552 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.206683 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.210601 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.210606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.210610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.210638 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.214192 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.214985 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.214990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.214993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.215182 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.215644 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.215647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.215649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.216109 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.216219 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.216623 (+ 404us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.216889 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":45}
I20260809 05:21:52.223080 323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Bootstrap starting.
I20260809 05:21:52.227612 323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:52.228961 323 log.cc:824] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:52.232026 323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: No bootstrap required, opened a new log
I20260809 05:21:52.243983 323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } }
I20260809 05:21:52.244345 323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:52.244547 323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 236e18c47d8e4d8aa3e193c7f7f4f452, State: Initialized, Role: FOLLOWER
I20260809 05:21:52.245285 323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } }
I20260809 05:21:52.246548 335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } } }
I20260809 05:21:52.246969 335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:52.247664 323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: configured and running, proceeding with master startup.
W20260809 05:21:52.262419 347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:21:52.262612 347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:21:52.266350 32553 master_runner.cc:428] Detected that this master 236e18c47d8e4d8aa3e193c7f7f4f452 is joining an existing cluster
I20260809 05:21:52.266481 32553 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.1.124:42515
I20260809 05:21:52.284981 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 31812
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.65@KRBTEST.COM
I20260809 05:21:52.306604 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.273781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.22.1.126:41321)
0809 05:21:52.273977 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.273984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.280023 (+ 6039us) server_negotiation.cc:407] Connection header received
0809 05:21:52.280658 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.280666 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.280765 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.280841 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.285467 (+ 4626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.286659 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.292847 (+ 6188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.293739 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.293847 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.297311 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.297319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.297324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.297366 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.302187 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.303151 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.303157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.303161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.303407 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.304296 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.304302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.304306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.304780 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.305230 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.305916 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.306175 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:21:52.307906 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.273611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43412)
0809 05:21:52.279694 (+ 6083us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.279714 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.280382 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.281127 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.281134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.283427 (+ 2293us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.285247 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.285264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.287319 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.287325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.292668 (+ 5343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.292680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.294297 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.294307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.294538 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.295513 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.295544 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.297014 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.302354 (+ 5340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.302358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.302362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.303005 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.303815 (+ 810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.303820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.303824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.304150 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.305319 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.305325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.305575 (+ 250us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.306896 (+ 1321us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.307309 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5933,"mutex_wait_us":3,"thread_start_us":7551,"threads_started":1}
I20260809 05:21:52.315680 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.286440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.22.1.65:40229)
0809 05:21:52.287231 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.287241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.287276 (+ 35us) server_negotiation.cc:407] Connection header received
0809 05:21:52.287350 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.287358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.287660 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.290118 (+ 2458us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.290575 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.292650 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.294996 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.296221 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.303512 (+ 7291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.303973 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.303991 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.304014 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.304092 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.309205 (+ 5113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.310247 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.310257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.310260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.310454 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.312013 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.312020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.312023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.312560 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.314462 (+ 1902us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.314678 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.314999 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":416,"threads_started":1}
I20260809 05:21:52.317839 31980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:52.318447 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.279944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.22.1.124:42515)
0809 05:21:52.280226 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.280231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.282955 (+ 2724us) server_negotiation.cc:407] Connection header received
0809 05:21:52.283525 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.283534 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.283613 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.283683 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.290771 (+ 7088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.292264 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.297173 (+ 4909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.297907 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.298003 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.302552 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.302559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.302563 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.302604 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.307020 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.309874 (+ 2854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.309879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.309882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.310188 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.316289 (+ 6101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.316294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.316297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.316857 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.317094 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.317685 (+ 591us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.317902 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":115}
I20260809 05:21:52.319365 308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.273098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.22.1.125:42903)
0809 05:21:52.277891 (+ 4793us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.277899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.281979 (+ 4080us) server_negotiation.cc:407] Connection header received
0809 05:21:52.282350 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.282358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.282440 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.282515 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.291285 (+ 8770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.292405 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.298973 (+ 6568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.299893 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.300012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.310483 (+ 10471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.310492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.310497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.310542 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.315418 (+ 4876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.316337 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.316343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.316346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.316511 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.316993 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.316998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.317001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.317503 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.317628 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.318468 (+ 840us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.318757 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4617}
I20260809 05:21:52.320497 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.279936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46096)
0809 05:21:52.282642 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.282660 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.283375 (+ 715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.283909 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.283914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.285781 (+ 1867us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.290279 (+ 4498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.290295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.292959 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.292965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.297017 (+ 4052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.297027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.298408 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.298414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.298594 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.300997 (+ 2403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.301022 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.302305 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.308362 (+ 6057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.308367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.308371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.309775 (+ 1404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.311412 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.311417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.311420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.313055 (+ 1635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.317174 (+ 4119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.317180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.317429 (+ 249us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.319262 (+ 1833us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.319633 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2541,"thread_start_us":2035,"threads_started":1}
I20260809 05:21:52.320709 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.273703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38880)
0809 05:21:52.281701 (+ 7998us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.281721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.282176 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.282738 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.282744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.288582 (+ 5838us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.290588 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.290603 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.292608 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.292614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.295223 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.295239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.300353 (+ 5114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.300361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.300559 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.303633 (+ 3074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.303657 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.309620 (+ 5963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.315600 (+ 5980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.315607 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.315610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.316089 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.316669 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.316674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.316677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.316866 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.317698 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.317702 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.318076 (+ 374us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.319674 (+ 1598us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.320183 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7818,"thread_start_us":5542,"threads_started":1}
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252909, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.126@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
W20260809 05:21:52.343209 32715 tablet.cc:2369] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:52.344157 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.315956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.22.1.124:42515)
0809 05:21:52.318684 (+ 2728us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.318689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.318706 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:21:52.318792 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.318797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.318965 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.319071 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.325313 (+ 6242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.326229 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.328351 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.329398 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.329552 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.335150 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.335158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.335163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.335220 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.340196 (+ 4976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.341443 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.341449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.341453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.341624 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.342129 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.342134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.342137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.342611 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.342754 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.343440 (+ 686us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.343727 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2628,"thread_start_us":12325,"threads_started":1}
I20260809 05:21:52.344210 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.315233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46112)
0809 05:21:52.315638 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.315659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.315764 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.319110 (+ 3346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.319116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.320005 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.320278 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.320294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.326406 (+ 6112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.326413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.328111 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.328136 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.329611 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.329619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.329899 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.330770 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.330796 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.334878 (+ 4082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.340393 (+ 5515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.340400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.340404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.341295 (+ 891us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.341759 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.341765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.341769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.341995 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.342789 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.342794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.343029 (+ 235us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.343184 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.343523 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230}
I20260809 05:21:52.346688 31714 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.22.1.124:42515: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
member_type: NON_VOTER
last_known_addr {
host: "127.22.1.124"
port: 42515
}
attrs {
promote: true
}
}
dest_uuid: "b5ad151ceb04438cb667b7907a660c9a"
cas_config_opid_index: 5
I20260809 05:21:52.348001 31714 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } }
I20260809 05:21:52.350222 301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.354821 301 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.22.1.124:42515
I20260809 05:21:52.356211 32629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 05:21:52.356361 32752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:52.356956 32553 master.cc:537] Master@127.22.1.124:42515 shutting down...
I20260809 05:21:52.362639 32752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260809 05:21:52.380952 368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260809 05:21:52.381994 368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:52.382722 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.383104 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:52.395793 32553 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:52.396232 32553 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:52.396430 32553 tablet_replica.cc:330] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: stopping tablet replica
I20260809 05:21:52.403964 32553 master.cc:552] Master@127.22.1.124:42515 shutdown complete.
I20260809 05:21:52.404057 32553 master_runner.cc:305] Clearing existing system tablet
I20260809 05:21:52.404456 32553 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:52.404574 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:52.409826 32553 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260809 05:21:52.410563 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.409923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46124)
0809 05:21:52.410091 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.410272 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260809 05:21:52.411190 31692 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a -> Peer 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124:42515): Couldn't send request to peer 236e18c47d8e4d8aa3e193c7f7f4f452. Status: Network error: Client connection negotiation failed: client connection to 127.22.1.124:42515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 05:21:52.411444 32629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.412366 377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:52.416911 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: NON_VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: true } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.417518 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: LEARNER
I20260809 05:21:52.417945 369 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: attempting to promote NON_VOTER 19299c1b7af24eb4af96c113298b06e5 to VOTER
I20260809 05:21:52.417945 32553 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:52.418221 32553 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260809 05:21:52.418393 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
format_stamp: "Formatted at 2026-08-09 05:21:52 on dist-test-slave-rmhg"
server_key: "56271f1010d0315db20b2622cbfa4e4a"
server_key_iv: "549a8f0ab65710938536c560fdf9a9e1"
server_key_version: "encryptionkey@0"
I20260809 05:21:52.418883 369 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } }
I20260809 05:21:52.421952 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.420743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46136)
0809 05:21:52.421603 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.421708 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
I20260809 05:21:52.423363 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.422869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46150)
0809 05:21:52.423096 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.423195 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:42515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260809 05:21:52.423637 31692 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a -> Peer 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124:42515): Couldn't send request to peer 236e18c47d8e4d8aa3e193c7f7f4f452. Status: Network error: Client connection negotiation failed: client connection to 127.22.1.124:42515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 05:21:52.424220 32629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 LEARNER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260809 05:21:52.424990 32553 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:52.425047 368 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260809 05:21:52.428512 301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.433595 32629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.434104 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.434464 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:52.435020 367 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.435411 367 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:52.438529 32667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.438954 32667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:52.439733 32553 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260809 05:21:52.440021 32553 log.cc:1196] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260809 05:21:52.440634 32553 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Deleting consensus metadata
I20260809 05:21:52.440886 32553 master_runner.cc:315] Copying system tablet from 127.22.1.126:41321
I20260809 05:21:52.445732 32553 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: tablet copy: Beginning tablet copy session from remote peer at address 127.22.1.126:41321
I20260809 05:21:52.462623 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.446619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43420)
0809 05:21:52.447577 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.447595 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.447764 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.448225 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.448231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.448803 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:52.449037 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.449049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.450362 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.450368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.451629 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.451645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.452668 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.452674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.452942 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.453787 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:52.453807 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:52.455120 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:52.459373 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.459379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.459383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.459829 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.460322 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:52.460327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:52.460330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:52.460488 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:52.461111 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:52.461117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:52.461407 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.461551 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.461930 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":463,"threads_started":1}
I20260809 05:21:52.462651 329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.446613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.22.1.126:41321)
0809 05:21:52.446868 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.446875 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.447703 (+ 828us) server_negotiation.cc:407] Connection header received
0809 05:21:52.447979 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.447984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.448063 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.448169 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:52.449219 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.450196 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.451774 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.452491 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.452642 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.455386 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:52.455393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:52.455397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:52.455435 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:52.459196 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.459974 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.459980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.459984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.460191 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:52.460601 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:52.460605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:52.460607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:52.460984 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:52.461127 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.461740 (+ 613us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.461967 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260809 05:21:52.463687 31741 tablet_copy_service.cc:140] P b5ad151ceb04438cb667b7907a660c9a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 236e18c47d8e4d8aa3e193c7f7f4f452 ({username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:43420)
I20260809 05:21:52.463945 31741 tablet_copy_service.cc:161] P b5ad151ceb04438cb667b7907a660c9a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 236e18c47d8e4d8aa3e193c7f7f4f452 at {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:43420: session id = 236e18c47d8e4d8aa3e193c7f7f4f452-00000000000000000000000000000000
I20260809 05:21:52.467514 31741 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Tablet Copy: opened 0 blocks and 1 log segments
I20260809 05:21:52.469519 32553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260809 05:21:52.473873 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 31812
I20260809 05:21:52.476773 32553 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: tablet copy: Starting download of 0 data blocks...
I20260809 05:21:52.477097 32553 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: tablet copy: Starting download of 1 WAL segments...
I20260809 05:21:52.479576 32553 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260809 05:21:52.485009 31741 tablet_copy_service.cc:342] P b5ad151ceb04438cb667b7907a660c9a: Request end of tablet copy session 236e18c47d8e4d8aa3e193c7f7f4f452-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:43420
I20260809 05:21:52.485217 31741 tablet_copy_service.cc:433] P b5ad151ceb04438cb667b7907a660c9a: ending tablet copy session 236e18c47d8e4d8aa3e193c7f7f4f452-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 236e18c47d8e4d8aa3e193c7f7f4f452
I20260809 05:21:52.495494 32553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:52.497997 385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.502832 386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.509084 391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:52.509352 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:40229
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:41321
--webserver_port=46311
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:21:52.518674 32553 server_base.cc:1034] running on GCE node
I20260809 05:21:52.519086 32553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:52.519783 32553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:52.520989 32553 hybrid_clock.cc:648] HybridClock initialized: now 1786252912520940 us; error 43 us; skew 500 ppm
I20260809 05:21:52.521301 32553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:52 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252912, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:52.523591 32553 init.cc:377] Logged in from keytab as oryx/127.22.1.124@KRBTEST.COM (short username oryx)
I20260809 05:21:52.526166 32553 webserver.cc:460] Webserver started at http://127.22.1.124:35233/ using document root <none> and password file <none>
I20260809 05:21:52.526500 32553 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:52.526602 32553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:52.529877 32553 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260809 05:21:52.531900 404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:52.532325 32553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260809 05:21:52.532500 32553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "236e18c47d8e4d8aa3e193c7f7f4f452"
format_stamp: "Formatted at 2026-08-09 05:21:52 on dist-test-slave-rmhg"
server_key: "56271f1010d0315db20b2622cbfa4e4a"
server_key_iv: "549a8f0ab65710938536c560fdf9a9e1"
server_key_version: "encryptionkey@0"
I20260809 05:21:52.532760 32553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:52.561920 32553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:52.566498 32553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:52.598169 32553 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.124:42515
I20260809 05:21:52.598204 485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.124:42515 every 8 connection(s)
I20260809 05:21:52.599525 32553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
I20260809 05:21:52.605103 486 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:21:52.608882 486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Bootstrap starting.
I20260809 05:21:52.628095 486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:21:52.628654 486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Bootstrap complete.
I20260809 05:21:52.629798 486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } }
I20260809 05:21:52.630112 486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 236e18c47d8e4d8aa3e193c7f7f4f452, State: Initialized, Role: LEARNER
I20260809 05:21:52.630369 486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } }
I20260809 05:21:52.631140 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.631433 486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:52.631667 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: LEARNER
I20260809 05:21:52.639150 514 catalog_manager.cc:1259] Loaded cluster ID: 82aabcafccfd4eada520395c3df12c1c
I20260809 05:21:52.639252 514 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: loading cluster ID for follower catalog manager: success
I20260809 05:21:52.642756 514 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: acquiring CA information for follower catalog manager: success
I20260809 05:21:52.645124 514 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:21:52.815200 306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.805306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46152)
0809 05:21:52.805534 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:52.805558 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:52.805724 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:52.807648 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:52.807653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:52.807691 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:52.807904 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.807917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.809847 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.809853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.811990 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:52.812021 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.814087 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.814094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:52.814546 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.814588 (+ 42us) client_negotiation.cc:769] Sending connection context
0809 05:21:52.814696 (+ 108us) client_negotiation.cc:240] Negotiation successful
0809 05:21:52.814862 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 05:21:52.815485 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:52.805547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.22.1.124:42515)
0809 05:21:52.807296 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:52.807301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:52.807319 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:21:52.807385 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:52.807389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:52.807462 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:52.807570 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:52.808114 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.809675 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.812221 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:52.813882 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:52.814030 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:52.814955 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:52.815016 (+ 61us) server_negotiation.cc:299] Negotiation successful
0809 05:21:52.815149 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1647,"thread_start_us":1508,"threads_started":1}
I20260809 05:21:52.816704 449 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260809 05:21:52.817554 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ad151ceb04438cb667b7907a660c9a. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:52.818022 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: LEARNER
W20260809 05:21:52.954057 393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:52.954411 393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:52.954640 393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:52.964933 393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:52.965109 393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:52.965296 393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:21:52.977387 393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:40229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=46311
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:52.979079 393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:52.980551 393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:52.998175 523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.998159 524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:52.999110 528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:52.999698 393 server_base.cc:1034] running on GCE node
I20260809 05:21:53.000586 393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:53.001845 393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:53.003051 393 hybrid_clock.cc:648] HybridClock initialized: now 1786252913002990 us; error 59 us; skew 500 ppm
I20260809 05:21:53.003471 393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:53.007735 393 init.cc:377] Logged in from keytab as oryx/127.22.1.65@KRBTEST.COM (short username oryx)
I20260809 05:21:53.009865 393 webserver.cc:460] Webserver started at http://127.22.1.65:46311/ using document root <none> and password file <none>
I20260809 05:21:53.010818 393 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:53.010991 393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:53.016987 393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 05:21:53.020406 535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.022065 393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:21:53.022248 393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "97539a6b121c4f1aa9cef992d0948676"
format_stamp: "Formatted at 2026-08-09 05:21:49 on dist-test-slave-rmhg"
server_key: "413f2257548f6283df2dc5d16491ed9b"
server_key_iv: "31f7bc9d3fe5fa47c841d7fbfee8d00b"
server_key_version: "encryptionkey@0"
I20260809 05:21:53.023460 393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:53.050809 393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:53.056257 393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:53.056754 393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:53.058450 393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:53.060352 393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:53.060559 393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.060735 393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:53.060918 393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.142997 393 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:40229
I20260809 05:21:53.143313 718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:40229 every 8 connection(s)
I20260809 05:21:53.144865 393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:21:53.149890 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 393
I20260809 05:21:53.150288 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 32001
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.66@KRBTEST.COM
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:53.198689 730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.151493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.22.1.66:43763)
0809 05:21:53.168083 (+ 16590us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.168097 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.168128 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:21:53.168191 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.168197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.168441 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.168599 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.171856 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.174238 (+ 2382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.176541 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.177801 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.177931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.186955 (+ 9024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.186973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.186991 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.187032 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.192444 (+ 5412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.193613 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.193623 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.193627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.193895 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.195261 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.195266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.195269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.195845 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.197674 (+ 1829us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.197892 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.198202 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16392,"thread_start_us":16210,"threads_started":1}
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:53.200428 32191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:53.204020 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.149045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56491 (local address 127.22.1.124:42515)
0809 05:21:53.149297 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.149305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.170969 (+ 21664us) server_negotiation.cc:407] Connection header received
0809 05:21:53.176212 (+ 5243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.176220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.176318 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.176415 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.179534 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.181012 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.184964 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.185611 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.185689 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.193055 (+ 7366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.193063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.193068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.193112 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.197643 (+ 4531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.200244 (+ 2601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.200251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.200254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.200523 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.201401 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.201408 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.201412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.201835 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.201985 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.203181 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.203567 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 05:21:53.204963 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.171798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:59573 (local address 127.22.1.125:42903)
0809 05:21:53.172497 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.172502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.181337 (+ 8835us) server_negotiation.cc:407] Connection header received
0809 05:21:53.181786 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.181793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.181897 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.181983 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.186085 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.187444 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.190443 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.191178 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.191279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.196783 (+ 5504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.196790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.196795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.196835 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.201035 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.202220 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.202226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.202229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.202436 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.202943 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.202948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.202951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.203408 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.203538 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.204121 (+ 583us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.204388 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":450,"threads_started":1}
I20260809 05:21:53.206864 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.151989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.22.1.65:56491)
0809 05:21:53.170756 (+ 18767us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.170813 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.175850 (+ 5037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.176958 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.176972 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.178273 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.179255 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.179284 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.181321 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.181327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.184708 (+ 3381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.184725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.186938 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.186950 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.187277 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.188182 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.188215 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.192740 (+ 4525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.199157 (+ 6417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.199166 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.199184 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.200085 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.200679 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.200684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.200687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.201239 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.202077 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.202087 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.202657 (+ 570us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.204344 (+ 1687us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.204991 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18288,"mutex_wait_us":4,"thread_start_us":18042,"threads_started":1}
I20260809 05:21:53.207002 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.171538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.22.1.65:59573)
0809 05:21:53.181194 (+ 9656us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.181249 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.181606 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.182343 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.182348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.183528 (+ 1180us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.184261 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.184276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.187694 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.187699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.190254 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.190272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.191664 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.191671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.191856 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.192709 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.192726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.196492 (+ 3766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.201238 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.201243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.201247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.202061 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.202592 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.202598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.202602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.202809 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.203619 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.203624 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.203804 (+ 180us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.205190 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.205483 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9501,"thread_start_us":9823,"threads_started":1}
I20260809 05:21:53.208815 724 heartbeater.cc:347] Connected to a master server at 127.22.1.125:42903
I20260809 05:21:53.208858 719 heartbeater.cc:347] Connected to a master server at 127.22.1.124:42515
I20260809 05:21:53.209126 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.157923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:60141)
0809 05:21:53.179991 (+ 22068us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.180016 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.180130 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.185745 (+ 5615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.185750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.186658 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.187384 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.187398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.188851 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.188857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.192328 (+ 3471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.192343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.194535 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.194541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.194851 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.195597 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.195618 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.200178 (+ 4560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.205398 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.205402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.205406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.206045 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.206531 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.206536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.206539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.206770 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.207912 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.207916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.208100 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.208234 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.208539 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21883,"mutex_wait_us":53,"thread_start_us":21561,"threads_started":1}
I20260809 05:21:53.209268 719 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.209303 724 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.209880 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.158269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:60141 (local address 127.22.1.126:41321)
0809 05:21:53.185174 (+ 26905us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.185180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.185201 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:21:53.185258 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.185263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.185338 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.185443 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.187565 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.188685 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.193259 (+ 4574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.193946 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.194060 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.200441 (+ 6381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.200448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.200453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.200497 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.204966 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.206187 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.206193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.206196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.206400 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.206917 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.206923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.206927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.207712 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.207837 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.209053 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.209544 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26747,"thread_start_us":26582,"threads_started":1}
I20260809 05:21:53.210621 724 heartbeater.cc:511] Master 127.22.1.125:42903 requested a full tablet report, sending...
I20260809 05:21:53.211701 720 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:53.211877 720 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.212752 720 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:53.213133 719 heartbeater.cc:511] Master 127.22.1.124:42515 requested a full tablet report, sending...
I20260809 05:21:53.213189 32611 ts_manager.cc:194] Registered new tserver with Master: 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:53.214390 31714 ts_manager.cc:194] Re-registered known tserver with Master: 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:53.215269 432 ts_manager.cc:194] Registered new tserver with Master: 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:53.215752 31714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:60141
I20260809 05:21:53.224728 301 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: NON_VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: true } } }
I20260809 05:21:53.225425 301 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:53.227463 740 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: attempting to promote NON_VOTER 236e18c47d8e4d8aa3e193c7f7f4f452 to VOTER
I20260809 05:21:53.228233 740 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } }
I20260809 05:21:53.230408 449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 LEARNER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 05:21:53.230406 32629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Refusing update from remote peer b5ad151ceb04438cb667b7907a660c9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260809 05:21:53.231212 740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 05:21:53.231812 741 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260809 05:21:53.234722 301 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.235368 32629 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.239159 301 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.239559 301 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:53.239976 301 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.240285 301 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:53.242172 745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.242671 745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:53.250500 740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.251051 740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:53.251220 449 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.255692 492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5ad151ceb04438cb667b7907a660c9a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:53.255911 492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:53.308184 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 32001
I20260809 05:21:53.331459 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:43763
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:41321
--webserver_port=46271
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:53.666399 746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:53.666692 746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:53.666796 746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:53.674166 746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:53.674276 746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:53.674418 746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:21:53.684368 746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:43763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=46271
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:53.685977 746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:53.687317 746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:53.701263 755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:53.701349 757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:53.701287 759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:53.706925 746 server_base.cc:1034] running on GCE node
I20260809 05:21:53.707670 746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:53.708776 746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:53.710004 746 hybrid_clock.cc:648] HybridClock initialized: now 1786252913709935 us; error 56 us; skew 500 ppm
I20260809 05:21:53.710373 746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:53.713609 746 init.cc:377] Logged in from keytab as oryx/127.22.1.66@KRBTEST.COM (short username oryx)
I20260809 05:21:53.715202 746 webserver.cc:460] Webserver started at http://127.22.1.66:46271/ using document root <none> and password file <none>
I20260809 05:21:53.715930 746 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:53.716075 746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:53.721583 746 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 05:21:53.724287 767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.725734 746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:53.725891 746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "2e4efbab91644fa48b2e95d8054b790e"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "0d1e9d8061d586e0777f564cdc208a21"
server_key_iv: "1800046f2375481b4c84dd0a61bc67c4"
server_key_version: "encryptionkey@0"
I20260809 05:21:53.726852 746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:53.746302 746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:53.750448 746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:53.750762 746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:53.751739 746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:53.753082 746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:53.753203 746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.753295 746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:53.753353 746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:53.785573 746 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:43763
I20260809 05:21:53.785650 941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:43763 every 8 connection(s)
I20260809 05:21:53.787103 746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:21:53.795725 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 746
I20260809 05:21:53.796057 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 32205
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.67@KRBTEST.COM
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
Aug 09 05:21:53 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252913, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
I20260809 05:21:53.831846 947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.790108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49425 (local address 127.22.1.126:41321)
0809 05:21:53.802504 (+ 12396us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.802512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.806056 (+ 3544us) server_negotiation.cc:407] Connection header received
0809 05:21:53.808031 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.808037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.808120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.808290 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.811137 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.811822 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.814335 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.815080 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.815231 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.820853 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.820857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.820861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.820895 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.825707 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.828055 (+ 2348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.828061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.828065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.828340 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.828921 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.828927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.828930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.829395 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.829570 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.830738 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.831072 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12231,"thread_start_us":12439,"threads_started":1}
I20260809 05:21:53.832031 948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.790337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:35069 (local address 127.22.1.124:42515)
0809 05:21:53.802206 (+ 11869us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.802213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.805300 (+ 3087us) server_negotiation.cc:407] Connection header received
0809 05:21:53.808530 (+ 3230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.808536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.808623 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.808843 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.811172 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.811918 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.814727 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.815312 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.815435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.823725 (+ 8290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.823733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.823738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.823780 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.828326 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.829237 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.829243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.829246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.829502 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.830069 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.830074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.830077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.830536 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.830709 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.831345 (+ 636us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.831603 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11722,"thread_start_us":11608,"threads_started":1}
I20260809 05:21:53.833016 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.790908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:49425)
0809 05:21:53.805807 (+ 14899us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.805834 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.807704 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.808549 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.808561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.809770 (+ 1209us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.810958 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.810987 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.811996 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.812001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.814201 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.814216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.815337 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.815345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.815626 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.816572 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.816607 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.820612 (+ 4005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.826984 (+ 6372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.826993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.827011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.827883 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.828512 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.828516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.828520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.828805 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.829613 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.829621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.830234 (+ 613us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.830648 (+ 414us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.831312 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14736,"mutex_wait_us":271,"spinlock_wait_cycles":13568,"thread_start_us":13122,"threads_started":1}
I20260809 05:21:53.833082 950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.790908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.22.1.66:35069)
0809 05:21:53.804817 (+ 13909us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.804882 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.807694 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.809104 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.809108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.809769 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.810972 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.810986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.813319 (+ 2333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.813325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.814600 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.814611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.815998 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.816004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.816160 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.818372 (+ 2212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.818394 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.823454 (+ 5060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.828501 (+ 5047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.828506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.828509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.829091 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.829657 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.829662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.829665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.829944 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.830769 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.830773 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.830929 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.831057 (+ 128us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.832511 (+ 1454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13513,"mutex_wait_us":1279,"thread_start_us":14384,"threads_started":1}
I20260809 05:21:53.833544 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.806053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:42011 (local address 127.22.1.125:42903)
0809 05:21:53.806524 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.806530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.807347 (+ 817us) server_negotiation.cc:407] Connection header received
0809 05:21:53.808030 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.808036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.808129 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.808390 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.812087 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.813125 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.818407 (+ 5282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.819320 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.819462 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.824827 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.824835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.824840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.824884 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.829911 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.830867 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.830872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.830876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.831046 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.831639 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.831645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.831648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.832125 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.832252 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.832789 (+ 537us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.833026 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":224,"threads_started":1}
I20260809 05:21:53.834772 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.806514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.22.1.66:42011)
0809 05:21:53.807119 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:53.807145 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:53.807703 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:53.808601 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:53.808605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:53.809878 (+ 1273us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:53.810963 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.810987 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.816339 (+ 5352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.816345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.818204 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:53.818221 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.819856 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.819866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:53.820203 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.820817 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:53.820840 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:53.824515 (+ 3675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:53.830070 (+ 5555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.830074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.830078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.830744 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.831175 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:53.831180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:53.831183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:53.831514 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:53.832332 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:53.832336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:53.832518 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 05:21:53.833779 (+ 1261us) client_negotiation.cc:240] Negotiation successful
0809 05:21:53.834128 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":27,"spinlock_wait_cycles":46464,"thread_start_us":296,"threads_started":1}
I20260809 05:21:53.835228 943 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:53.835685 943 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.835969 944 heartbeater.cc:347] Connected to a master server at 127.22.1.125:42903
I20260809 05:21:53.836125 944 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.836663 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:53.797903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.22.1.67:40243)
0809 05:21:53.803147 (+ 5244us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:53.803158 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:53.803905 (+ 747us) server_negotiation.cc:407] Connection header received
0809 05:21:53.804143 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:53.804154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:53.804480 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:53.804913 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:53.807328 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.808752 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.810436 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:53.812119 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:53.812236 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:53.821358 (+ 9122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:53.821374 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:53.821395 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:53.821449 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:53.826815 (+ 5366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.828579 (+ 1764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.828588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.828591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.829613 (+ 1022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:53.830411 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:53.830415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:53.830418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:53.832988 (+ 2570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:53.835723 (+ 2735us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:53.835885 (+ 162us) server_negotiation.cc:299] Negotiation successful
0809 05:21:53.836159 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5058,"thread_start_us":4872,"threads_started":1}
I20260809 05:21:53.836762 942 heartbeater.cc:347] Connected to a master server at 127.22.1.124:42515
I20260809 05:21:53.836982 942 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:53.837297 944 heartbeater.cc:511] Master 127.22.1.125:42903 requested a full tablet report, sending...
I20260809 05:21:53.838023 942 heartbeater.cc:511] Master 127.22.1.124:42515 requested a full tablet report, sending...
I20260809 05:21:53.838568 943 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:53.838712 32392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:53.839437 32611 ts_manager.cc:194] Registered new tserver with Master: 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:53.840196 432 ts_manager.cc:194] Registered new tserver with Master: 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:53.841235 31714 ts_manager.cc:194] Re-registered known tserver with Master: 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:53.842402 31714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49425
I20260809 05:21:53.924962 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 32205
I20260809 05:21:53.949589 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:40243
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:41321
--webserver_port=35841
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:21:54.219020 720 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
W20260809 05:21:54.286504 961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:54.286777 961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:54.286880 961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:54.294087 961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:54.294198 961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:54.294317 961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:21:54.303831 961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:40243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=35841
--tserver_master_addrs=127.22.1.126:41321,127.22.1.125:42903,127.22.1.124:42515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:54.305119 961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:54.306478 961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:54.319952 972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:54.319977 969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:54.320473 967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:54.321317 961 server_base.cc:1034] running on GCE node
I20260809 05:21:54.321940 961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:54.322921 961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:54.324177 961 hybrid_clock.cc:648] HybridClock initialized: now 1786252914324118 us; error 46 us; skew 500 ppm
I20260809 05:21:54.324543 961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:54 dist-test-slave-rmhg krb5kdc[31652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252914, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:54.327872 961 init.cc:377] Logged in from keytab as oryx/127.22.1.67@KRBTEST.COM (short username oryx)
I20260809 05:21:54.329608 961 webserver.cc:460] Webserver started at http://127.22.1.67:35841/ using document root <none> and password file <none>
I20260809 05:21:54.330322 961 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:54.330466 961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:54.337066 961 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260809 05:21:54.340188 985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:54.341811 961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:21:54.341984 961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "d08f2e0339164b1a8d51add7961ad3a5"
format_stamp: "Formatted at 2026-08-09 05:21:50 on dist-test-slave-rmhg"
server_key: "bdb596c2cfcc431b8e2eb09e57382247"
server_key_iv: "add27c84177b733a5ddb036475ca8fdc"
server_key_version: "encryptionkey@0"
I20260809 05:21:54.343025 961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:54.379216 961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:54.383996 961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:54.384299 961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:54.385365 961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:21:54.386730 961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:21:54.386817 961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:54.386900 961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:21:54.386953 961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:54.417487 961 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40243
I20260809 05:21:54.417568 1167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40243 every 8 connection(s)
I20260809 05:21:54.418984 961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:21:54.424075 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 961
Aug 09 05:21:54 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252914, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
Aug 09 05:21:54 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252914, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.125@KRBTEST.COM
Aug 09 05:21:54 dist-test-slave-rmhg krb5kdc[31652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252914, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.124@KRBTEST.COM
I20260809 05:21:54.461930 1178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.422510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:34441)
0809 05:21:54.429895 (+ 7385us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.429963 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.432799 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.434163 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.434176 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.435320 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:54.436282 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.436309 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.438030 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.438036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.440370 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.440392 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.444359 (+ 3967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.444366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.444818 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.445634 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:54.445669 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:54.448999 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:54.454550 (+ 5551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.454561 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.454580 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.455003 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.455453 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.455458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.455462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.455751 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.458379 (+ 2628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:54.458389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:54.458945 (+ 556us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.459293 (+ 348us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.460010 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7025,"mutex_wait_us":44,"thread_start_us":10695,"threads_started":1}
I20260809 05:21:54.463428 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.421778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:34441 (local address 127.22.1.126:41321)
0809 05:21:54.428975 (+ 7197us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.428982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.433485 (+ 4503us) server_negotiation.cc:407] Connection header received
0809 05:21:54.433572 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.433578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.433670 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.435135 (+ 1465us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.436512 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.437807 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.440611 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.441344 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.441472 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.449402 (+ 7930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.449409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.449415 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.449477 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.454374 (+ 4897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.455147 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.455153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.455156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.455326 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.457658 (+ 2332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.457664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.457667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.458204 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.462475 (+ 4271us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.462629 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.462911 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7053,"thread_start_us":6936,"threads_started":1}
I20260809 05:21:54.465183 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.422510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.22.1.67:52161)
0809 05:21:54.433643 (+ 11133us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.433661 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.434131 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.434981 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.434985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.435563 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:54.441908 (+ 6345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.441925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.443078 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.443082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.444069 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.444080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.445926 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.445936 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.446104 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.446734 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:54.446757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:54.450324 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:54.460353 (+ 10029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.460358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.460361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.461221 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.462220 (+ 999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.462226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.462230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.462450 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.463908 (+ 1458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:54.463915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:54.464123 (+ 208us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.464263 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.464599 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10957,"mutex_wait_us":5893,"spinlock_wait_cycles":32640,"thread_start_us":6729,"threads_started":1}
I20260809 05:21:54.466771 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.422120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:52161 (local address 127.22.1.125:42903)
0809 05:21:54.432113 (+ 9993us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.432118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.433798 (+ 1680us) server_negotiation.cc:407] Connection header received
0809 05:21:54.434280 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.434286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.434367 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.434728 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.442141 (+ 7413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.442898 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.444223 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.445560 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.445688 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.450566 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.450574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.450580 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.450629 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.460070 (+ 9441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.461415 (+ 1345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.461421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.461424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.461652 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.462663 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.462668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.462672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.463150 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.465812 (+ 2662us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.465947 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.466227 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9875,"thread_start_us":9771,"threads_started":1}
I20260809 05:21:54.468441 1170 heartbeater.cc:347] Connected to a master server at 127.22.1.125:42903
I20260809 05:21:54.468870 1170 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:54.470276 1170 heartbeater.cc:511] Master 127.22.1.125:42903 requested a full tablet report, sending...
I20260809 05:21:54.471347 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.430886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:40467 (local address 127.22.1.124:42515)
0809 05:21:54.441767 (+ 10881us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.441774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.443016 (+ 1242us) server_negotiation.cc:407] Connection header received
0809 05:21:54.443105 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.443112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.443204 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.443510 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.445205 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.446369 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.448546 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.449278 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.449383 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.456543 (+ 7160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.456552 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.456558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.456610 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.466692 (+ 10082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.467688 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.467694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.467700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.467918 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.468482 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.468488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.468492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.469086 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.469258 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.470493 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.470785 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10719,"thread_start_us":10889,"threads_started":1}
I20260809 05:21:54.472162 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.433430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.22.1.67:40467)
0809 05:21:54.442363 (+ 8933us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.442386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.442840 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.443726 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.443732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.444472 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:21:54.444969 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.444982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.446552 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.446557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.448328 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.448350 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.449722 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.449730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.450069 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.451167 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:21:54.451193 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:21:54.456188 (+ 4995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:21:54.466980 (+ 10792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.466985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.466989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.467500 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.468063 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:21:54.468069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:21:54.468073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:21:54.468335 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:21:54.469344 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:21:54.469350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:21:54.469535 (+ 185us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.469708 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.470155 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8802,"mutex_wait_us":415,"thread_start_us":8228,"threads_started":1}
I20260809 05:21:54.472544 32611 ts_manager.cc:194] Registered new tserver with Master: d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243)
I20260809 05:21:54.474220 1168 heartbeater.cc:347] Connected to a master server at 127.22.1.124:42515
I20260809 05:21:54.474409 1168 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:54.475174 1168 heartbeater.cc:511] Master 127.22.1.124:42515 requested a full tablet report, sending...
I20260809 05:21:54.476688 432 ts_manager.cc:194] Registered new tserver with Master: d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243)
I20260809 05:21:54.476809 1169 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41321
I20260809 05:21:54.477052 1169 heartbeater.cc:464] Registering TS with master...
I20260809 05:21:54.478140 1169 heartbeater.cc:511] Master 127.22.1.126:41321 requested a full tablet report, sending...
I20260809 05:21:54.479446 31714 ts_manager.cc:194] Re-registered known tserver with Master: d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243)
I20260809 05:21:54.480787 31714 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:34441
I20260809 05:21:54.503053 1173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.456055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.22.1.126:41321)
0809 05:21:54.463852 (+ 7797us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.463858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.481341 (+ 17483us) server_negotiation.cc:407] Connection header received
0809 05:21:54.481406 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.481412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.481489 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.481575 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.486040 (+ 4465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.486938 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.488589 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.489437 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.489563 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.494391 (+ 4828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.494396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.494399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.494431 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.499013 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.500728 (+ 1715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.500735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.500739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.500991 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.501506 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.501511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.501514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.501929 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.502207 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.502508 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.502703 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7617,"thread_start_us":19444,"threads_started":1}
I20260809 05:21:54.504698 1175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.452856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.22.1.125:42903)
0809 05:21:54.468575 (+ 15719us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.468580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.479445 (+ 10865us) server_negotiation.cc:407] Connection header received
0809 05:21:54.479713 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.479719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.479900 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.480007 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.484892 (+ 4885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.486481 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.489240 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.490053 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.490173 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.494184 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.494194 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.494199 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.494244 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.499568 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.500686 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.500690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.500692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.500864 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.501344 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.501351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.501355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.501883 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.502193 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.503931 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.504239 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15565,"thread_start_us":21661,"threads_started":1}
I20260809 05:21:54.507454 1182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.481239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.22.1.124:42515)
0809 05:21:54.481821 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.481828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.483627 (+ 1799us) server_negotiation.cc:407] Connection header received
0809 05:21:54.483938 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.483943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.484087 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.484179 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:54.488697 (+ 4518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.490027 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.492910 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.494022 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.494137 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.498798 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:54.498806 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:54.498810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:54.498861 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:54.503666 (+ 4805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.504713 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.504719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.504722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.504953 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:54.505492 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:54.505498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:54.505501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:54.505917 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:54.506168 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.506779 (+ 611us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.507065 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452}
I20260809 05:21:54.508503 31714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:54.520000 31714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:54.567847 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.547123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40243 (local address 127.0.0.1:37028)
0809 05:21:54.549954 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.549978 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.550466 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.555253 (+ 4787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.555260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.555299 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:54.555818 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.555834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.560155 (+ 4321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.560161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.563715 (+ 3554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.563735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.566766 (+ 3031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.566773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.567005 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.567043 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.567147 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.567297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2618,"thread_start_us":2434,"threads_started":1}
I20260809 05:21:54.568353 1219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.547604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37028 (local address 127.22.1.67:40243)
0809 05:21:54.554170 (+ 6566us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.554181 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.554225 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:21:54.554361 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.554371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.554698 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.555244 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:54.557313 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.559982 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.563914 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.566101 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.566232 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.566505 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.567262 (+ 757us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.567479 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6324,"thread_start_us":6122,"threads_started":1}
I20260809 05:21:54.570477 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.547793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.22.1.66:43763)
0809 05:21:54.556493 (+ 8700us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.556504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.558231 (+ 1727us) server_negotiation.cc:407] Connection header received
0809 05:21:54.558415 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.558425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.558772 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.559134 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:54.561574 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.563184 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.566037 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.567499 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.567688 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.567935 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.569904 (+ 1969us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.570119 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8439,"thread_start_us":8304,"threads_started":1}
I20260809 05:21:54.573827 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.547604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:43763 (local address 127.0.0.1:57976)
0809 05:21:54.557477 (+ 9873us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.557720 (+ 243us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.558073 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.559451 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.559458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.559489 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:54.561209 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.561227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.563330 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.563336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.565502 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.565516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.568551 (+ 3035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.568562 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.568925 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.569120 (+ 195us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.569239 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.569401 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9712,"mutex_wait_us":144,"thread_start_us":9470,"threads_started":1}
I20260809 05:21:54.573342 1068 tablet_service.cc:1467] Processing CreateTablet for tablet bf894a75f463407ca04a79b8d5c50d8d (DEFAULT_TABLE table=test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:54.575531 1068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf894a75f463407ca04a79b8d5c50d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:54.578874 852 tablet_service.cc:1467] Processing CreateTablet for tablet bf894a75f463407ca04a79b8d5c50d8d (DEFAULT_TABLE table=test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:54.579986 1222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.557683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40229 (local address 127.0.0.1:45912)
0809 05:21:54.562143 (+ 4460us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:54.562169 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:54.562449 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:54.564485 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:54.564490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:54.564525 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:54.565744 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.565760 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.567854 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.567859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.575160 (+ 7301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:54.575185 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.578450 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.578460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:54.578817 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.578860 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 05:21:54.578960 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 05:21:54.579125 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4222,"mutex_wait_us":24,"thread_start_us":3552,"threads_started":1}
I20260809 05:21:54.580731 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf894a75f463407ca04a79b8d5c50d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:54.581112 1221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:54.557935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.22.1.65:40229)
0809 05:21:54.560818 (+ 2883us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:54.560827 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:54.563352 (+ 2525us) server_negotiation.cc:407] Connection header received
0809 05:21:54.563517 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:54.563525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:54.563759 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:54.564105 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:54.565983 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.567646 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.575521 (+ 7875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:54.577771 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:54.577932 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:54.578169 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:54.580341 (+ 2172us) server_negotiation.cc:299] Negotiation successful
0809 05:21:54.580581 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2718,"thread_start_us":2572,"threads_started":1}
I20260809 05:21:54.586292 610 tablet_service.cc:1467] Processing CreateTablet for tablet bf894a75f463407ca04a79b8d5c50d8d (DEFAULT_TABLE table=test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:54.588315 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf894a75f463407ca04a79b8d5c50d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:54.608073 1224 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Bootstrap starting.
I20260809 05:21:54.615615 1224 tablet_bootstrap.cc:654] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:54.618435 1227 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Bootstrap starting.
I20260809 05:21:54.618599 1224 log.cc:824] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:54.619938 1230 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Bootstrap starting.
I20260809 05:21:54.624927 1227 tablet_bootstrap.cc:654] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:54.625339 1230 tablet_bootstrap.cc:654] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:54.627099 1230 log.cc:824] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:54.627219 1227 log.cc:824] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:54.630013 1224 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: No bootstrap required, opened a new log
I20260809 05:21:54.630383 1224 ts_tablet_manager.cc:1397] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.004s
I20260809 05:21:54.631937 1230 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: No bootstrap required, opened a new log
I20260809 05:21:54.632216 1227 tablet_bootstrap.cc:492] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: No bootstrap required, opened a new log
I20260809 05:21:54.632383 1230 ts_tablet_manager.cc:1397] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260809 05:21:54.632519 1227 ts_tablet_manager.cc:1397] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260809 05:21:54.640475 1224 raft_consensus.cc:348] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.641112 1224 raft_consensus.cc:374] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:54.641348 1224 raft_consensus.cc:729] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08f2e0339164b1a8d51add7961ad3a5, State: Initialized, Role: FOLLOWER
I20260809 05:21:54.642195 1224 consensus_queue.cc:260] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.644042 1169 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
I20260809 05:21:54.646695 1230 raft_consensus.cc:348] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.647387 1230 raft_consensus.cc:374] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:54.647526 1230 raft_consensus.cc:729] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97539a6b121c4f1aa9cef992d0948676, State: Initialized, Role: FOLLOWER
I20260809 05:21:54.648452 1230 consensus_queue.cc:260] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.649338 1224 ts_tablet_manager.cc:1428] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260809 05:21:54.650691 1227 raft_consensus.cc:348] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.651258 1227 raft_consensus.cc:374] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:54.651368 1227 raft_consensus.cc:729] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4efbab91644fa48b2e95d8054b790e, State: Initialized, Role: FOLLOWER
I20260809 05:21:54.654246 1230 ts_tablet_manager.cc:1428] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Time spent starting tablet: real 0.022s user 0.009s sys 0.013s
I20260809 05:21:54.654904 1227 consensus_queue.cc:260] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:54.656782 943 heartbeater.cc:503] Master 127.22.1.126:41321 was elected leader, sending a full tablet report...
I20260809 05:21:54.660112 1227 ts_tablet_manager.cc:1428] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Time spent starting tablet: real 0.027s user 0.012s sys 0.006s
W20260809 05:21:54.671993 1172 tablet.cc:2369] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:54.678326 726 tablet.cc:2369] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:21:54.790942 946 tablet.cc:2369] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:21:55.071453 1241 raft_consensus.cc:482] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:55.071794 1241 raft_consensus.cc:504] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:55.073598 1241 leader_election.cc:288] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243), 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:55.091387 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.073998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:37391 (local address 127.22.1.67:40243)
0809 05:21:55.086435 (+ 12437us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.086442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.086468 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:21:55.086650 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.086656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.086732 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.086820 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:55.087299 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.088009 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.089743 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.090498 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.090622 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.090723 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.090994 (+ 271us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.091127 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12313,"thread_start_us":12575,"threads_started":1}
I20260809 05:21:55.091435 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.073909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40243 (local address 127.22.1.65:37391)
0809 05:21:55.086247 (+ 12338us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:55.086277 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:55.086454 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:55.086859 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:55.086864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:55.086889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:55.087146 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:55.087157 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.088147 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.088151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:55.089606 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:55.089618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.090697 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.090703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:55.090835 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.090866 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 05:21:55.090947 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:21:55.091088 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12144,"mutex_wait_us":38,"thread_start_us":11956,"threads_started":1}
I20260809 05:21:55.092496 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf894a75f463407ca04a79b8d5c50d8d" candidate_uuid: "97539a6b121c4f1aa9cef992d0948676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08f2e0339164b1a8d51add7961ad3a5" is_pre_election: true
I20260809 05:21:55.092983 1095 raft_consensus.cc:2454] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97539a6b121c4f1aa9cef992d0948676 in term 0.
I20260809 05:21:55.093842 537 leader_election.cc:302] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97539a6b121c4f1aa9cef992d0948676, d08f2e0339164b1a8d51add7961ad3a5; no voters:
I20260809 05:21:55.094314 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.074121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:43763 (local address 127.22.1.65:58867)
0809 05:21:55.086459 (+ 12338us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:55.086479 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:55.086618 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:55.086935 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:55.086939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:55.086963 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:55.087294 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:55.087302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.088629 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.088633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:55.090939 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:55.090951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.093572 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.093579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:55.093721 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.093746 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 05:21:55.093833 (+ 87us) client_negotiation.cc:240] Negotiation successful
0809 05:21:55.093926 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12189,"mutex_wait_us":43,"thread_start_us":12025,"threads_started":1}
I20260809 05:21:55.094527 1241 raft_consensus.cc:2781] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:55.094616 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.074125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:58867 (local address 127.22.1.66:43763)
0809 05:21:55.086609 (+ 12484us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.086615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.086635 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:21:55.086745 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.086749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.086814 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.086896 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:55.087438 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.088482 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.091131 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.093186 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.093365 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.093505 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.093890 (+ 385us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.094076 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11628,"thread_start_us":11485,"threads_started":1}
I20260809 05:21:55.094702 1241 raft_consensus.cc:482] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:55.094815 1241 raft_consensus.cc:3037] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:55.095466 877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf894a75f463407ca04a79b8d5c50d8d" candidate_uuid: "97539a6b121c4f1aa9cef992d0948676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4efbab91644fa48b2e95d8054b790e" is_pre_election: true
I20260809 05:21:55.095927 877 raft_consensus.cc:2454] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97539a6b121c4f1aa9cef992d0948676 in term 0.
I20260809 05:21:55.097523 1241 raft_consensus.cc:504] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:55.098369 1241 leader_election.cc:288] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [CANDIDATE]: Term 1 election: Requested vote from peers d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243), 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763)
I20260809 05:21:55.098660 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf894a75f463407ca04a79b8d5c50d8d" candidate_uuid: "97539a6b121c4f1aa9cef992d0948676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08f2e0339164b1a8d51add7961ad3a5"
I20260809 05:21:55.098922 1095 raft_consensus.cc:3037] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:55.099223 877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf894a75f463407ca04a79b8d5c50d8d" candidate_uuid: "97539a6b121c4f1aa9cef992d0948676" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4efbab91644fa48b2e95d8054b790e"
I20260809 05:21:55.099485 877 raft_consensus.cc:3037] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:55.102144 1095 raft_consensus.cc:2454] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97539a6b121c4f1aa9cef992d0948676 in term 1.
I20260809 05:21:55.102645 537 leader_election.cc:302] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97539a6b121c4f1aa9cef992d0948676, d08f2e0339164b1a8d51add7961ad3a5; no voters:
I20260809 05:21:55.102946 1241 raft_consensus.cc:2781] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:55.103538 1241 raft_consensus.cc:686] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 1 LEADER]: Becoming Leader. State: Replica: 97539a6b121c4f1aa9cef992d0948676, State: Running, Role: LEADER
I20260809 05:21:55.103564 877 raft_consensus.cc:2454] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97539a6b121c4f1aa9cef992d0948676 in term 1.
I20260809 05:21:55.104337 1241 consensus_queue.cc:237] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:55.110950 31715 catalog_manager.cc:5495] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "97539a6b121c4f1aa9cef992d0948676" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } health_report { overall_health: HEALTHY } } }
I20260809 05:21:55.163923 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.157986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.22.1.65:40229)
0809 05:21:55.158475 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.158480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.158557 (+ 77us) server_negotiation.cc:407] Connection header received
0809 05:21:55.158761 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.158765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.158854 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.159048 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:55.159370 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.160159 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.161756 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.162174 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.162269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.162718 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:55.163278 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:55.163386 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.163471 (+ 85us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.163544 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260809 05:21:55.174144 877 raft_consensus.cc:1261] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Refusing update from remote peer 97539a6b121c4f1aa9cef992d0948676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:55.174234 1095 raft_consensus.cc:1261] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Refusing update from remote peer 97539a6b121c4f1aa9cef992d0948676: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:55.175333 1256 consensus_queue.cc:1035] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260809 05:21:55.175928 1241 consensus_queue.cc:1035] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:55.189456 1277 mvcc.cc:205] Tried to move back new op lower bound from 7316491940540760064 to 7316491940271927296. Current Snapshot: MvccSnapshot[applied={T|T < 7316491940540760064}]
I20260809 05:21:55.213537 31715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:55.213908 31715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:55.224920 31715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5ad151ceb04438cb667b7907a660c9a: Sending DeleteTablet for 3 replicas of tablet bf894a75f463407ca04a79b8d5c50d8d
I20260809 05:21:55.225814 852 tablet_service.cc:1514] Processing DeleteTablet for tablet bf894a75f463407ca04a79b8d5c50d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:55 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:57976
I20260809 05:21:55.226157 1068 tablet_service.cc:1514] Processing DeleteTablet for tablet bf894a75f463407ca04a79b8d5c50d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:55 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37028
I20260809 05:21:55.227320 849 tablet_service.cc:1514] Processing DeleteTablet for tablet bf894a75f463407ca04a79b8d5c50d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:55 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:57976
I20260809 05:21:55.227528 610 tablet_service.cc:1514] Processing DeleteTablet for tablet bf894a75f463407ca04a79b8d5c50d8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:55 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:45912
I20260809 05:21:55.231341 1296 tablet_replica.cc:330] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: stopping tablet replica
I20260809 05:21:55.232105 1296 raft_consensus.cc:2229] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:55.233008 1296 raft_consensus.cc:2258] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:55.233103 1297 tablet_replica.cc:330] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: stopping tablet replica
I20260809 05:21:55.233822 1297 raft_consensus.cc:2229] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:55.234748 1297 raft_consensus.cc:2258] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:55.235071 1296 ts_tablet_manager.cc:1905] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:55.236974 1297 ts_tablet_manager.cc:1905] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:55.241405 1296 ts_tablet_manager.cc:1918] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:55.242311 1300 tablet_replica.cc:330] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: stopping tablet replica
I20260809 05:21:55.243028 1300 raft_consensus.cc:2229] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:55.241632 1296 log.cc:1196] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/bf894a75f463407ca04a79b8d5c50d8d
I20260809 05:21:55.244817 1300 raft_consensus.cc:2258] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:55.246678 1297 ts_tablet_manager.cc:1918] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:55.246966 1297 log.cc:1196] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/bf894a75f463407ca04a79b8d5c50d8d
I20260809 05:21:55.247761 1297 ts_tablet_manager.cc:1939] T bf894a75f463407ca04a79b8d5c50d8d P d08f2e0339164b1a8d51add7961ad3a5: Deleting consensus metadata
I20260809 05:21:55.249439 1300 ts_tablet_manager.cc:1905] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:55.250079 31691 catalog_manager.cc:4841] TS d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243): tablet bf894a75f463407ca04a79b8d5c50d8d (table test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]) successfully deleted
I20260809 05:21:55.251626 1296 ts_tablet_manager.cc:1939] T bf894a75f463407ca04a79b8d5c50d8d P 2e4efbab91644fa48b2e95d8054b790e: Deleting consensus metadata
I20260809 05:21:55.253926 31694 catalog_manager.cc:4841] TS 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763): tablet bf894a75f463407ca04a79b8d5c50d8d (table test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]) successfully deleted
I20260809 05:21:55.254101 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 31667
I20260809 05:21:55.255447 31758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:55.264950 1300 ts_tablet_manager.cc:1918] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:55.265262 1300 log.cc:1196] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/bf894a75f463407ca04a79b8d5c50d8d
I20260809 05:21:55.265929 1300 ts_tablet_manager.cc:1939] T bf894a75f463407ca04a79b8d5c50d8d P 97539a6b121c4f1aa9cef992d0948676: Deleting consensus metadata
I20260809 05:21:55.270004 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.231154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.22.1.125:42903)
0809 05:21:55.242237 (+ 11083us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.242244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.248719 (+ 6475us) server_negotiation.cc:407] Connection header received
0809 05:21:55.248807 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.248812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.248909 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.249062 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:55.253812 (+ 4750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.255159 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.257417 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.257993 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.258143 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.263086 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:55.263092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:55.263095 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:55.263132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:55.266607 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.267676 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.267682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.267685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.267863 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.268289 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.268295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.268298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.268710 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:55.269101 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.269290 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.269529 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10769,"thread_start_us":10597,"threads_started":1}
I20260809 05:21:55.288261 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.246341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46170 (local address 127.22.1.124:42515)
0809 05:21:55.255167 (+ 8826us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.255172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.270007 (+ 14835us) server_negotiation.cc:407] Connection header received
0809 05:21:55.270776 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.270781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.270856 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.271263 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:55.273057 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.274299 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.275864 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.276694 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.276914 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.280358 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:55.280364 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:55.280368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:55.280401 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:55.284292 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.285127 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.285132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.285135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.285377 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.285839 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.285844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.285848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.286322 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:55.286573 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.287711 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.287892 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8649,"thread_start_us":8636,"threads_started":1}
I20260809 05:21:55.433995 31694 catalog_manager.cc:4841] TS 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229): tablet bf894a75f463407ca04a79b8d5c50d8d (table test-table [id=f3f3cf80a4fa406eb6b905c4382e00cc]) successfully deleted
W20260809 05:21:55.434497 31694 catalog_manager.cc:4808] TS 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763): delete failed for tablet bf894a75f463407ca04a79b8d5c50d8d because the tablet was not found. No further retry: Not found: Tablet not found: bf894a75f463407ca04a79b8d5c50d8d
I20260809 05:21:55.435472 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 31667
I20260809 05:21:55.487632 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.470466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.22.1.125:42903)
0809 05:21:55.470708 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.470714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.472167 (+ 1453us) server_negotiation.cc:407] Connection header received
0809 05:21:55.472487 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.472493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.472586 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.472651 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:55.474762 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.475709 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.477779 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.478468 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.478594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.481769 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:55.481774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:55.481778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:55.481818 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:55.484780 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.485527 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.485532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.485534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.485667 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.486122 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.486126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.486128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.486411 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:55.486670 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.487077 (+ 407us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.487295 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 05:21:55.489468 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:55.472242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.22.1.124:42515)
0809 05:21:55.472466 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:55.472472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:55.472529 (+ 57us) server_negotiation.cc:407] Connection header received
0809 05:21:55.472743 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:55.472748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:55.472830 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:55.472929 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:55.475205 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.476143 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.477625 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:55.478294 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:55.478433 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:55.482357 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:55.482365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:55.482370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:55.482416 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:55.486129 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.487025 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.487031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.487035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.487195 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:55.487767 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:55.487772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:55.487776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:55.488144 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:55.488368 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:55.488969 (+ 601us) server_negotiation.cc:299] Negotiation successful
0809 05:21:55.489194 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:21:56.195708 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.194844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:52103)
0809 05:21:56.195361 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.195506 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":262,"threads_started":1}
W20260809 05:21:56.196201 1169 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.208603 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.207777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:40245)
0809 05:21:56.208285 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.208429 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":378,"threads_started":1}
W20260809 05:21:56.209174 720 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.232940 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.232147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:35763)
0809 05:21:56.232618 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.232740 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":235,"threads_started":1}
W20260809 05:21:56.233393 943 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.723783 1321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5ad151ceb04438cb667b7907a660c9a)
I20260809 05:21:56.724071 1321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } }
I20260809 05:21:56.724363 1323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b5ad151ceb04438cb667b7907a660c9a)
I20260809 05:21:56.724627 1323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } }
I20260809 05:21:56.725780 1321 leader_election.cc:288] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321), 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125:42903)
I20260809 05:21:56.726095 1323 leader_election.cc:288] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321), 236e18c47d8e4d8aa3e193c7f7f4f452 (127.22.1.124:42515)
I20260809 05:21:56.737349 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.726232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43836)
0809 05:21:56.736749 (+ 10517us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.736951 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10377,"thread_start_us":10217,"threads_started":1}
W20260809 05:21:56.738134 32590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.741457 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.740289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43850)
0809 05:21:56.741140 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.741258 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":757,"threads_started":1}
W20260809 05:21:56.741705 32590 leader_election.cc:334] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321): Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.743455 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.726008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43826)
0809 05:21:56.742760 (+ 16752us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.743038 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16629,"mutex_wait_us":420,"thread_start_us":15525,"threads_started":1}
W20260809 05:21:56.743734 411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.747136 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.746716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43860)
0809 05:21:56.746910 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.746997 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260809 05:21:56.747469 411 leader_election.cc:334] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321): Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.749292 1326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.726153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.22.1.125:42903)
0809 05:21:56.742507 (+ 16354us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:56.742512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:56.742527 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:21:56.742579 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:56.742582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:56.742634 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:56.742719 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:56.743428 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.744698 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.747118 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.748236 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.748338 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.748419 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:56.748830 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 05:21:56.748938 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16242,"thread_start_us":16088,"threads_started":1}
I20260809 05:21:56.749357 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.726005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:38928)
0809 05:21:56.741706 (+ 15701us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.741726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:56.741873 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:56.742789 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:56.742795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:56.742824 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:56.743182 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.743201 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.744985 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.744991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.746955 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.746969 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.748409 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.748417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.748637 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.748678 (+ 41us) client_negotiation.cc:769] Sending connection context
0809 05:21:56.748782 (+ 104us) client_negotiation.cc:240] Negotiation successful
0809 05:21:56.748920 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15581,"mutex_wait_us":1,"thread_start_us":16486,"threads_started":1}
I20260809 05:21:56.749997 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.726464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:42515 (local address 127.0.0.1:46194)
0809 05:21:56.740519 (+ 14055us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.740541 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:56.740700 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:56.743731 (+ 3031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:56.743737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:56.743763 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:56.744144 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.744155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.745821 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.745828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.747743 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.747759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.749219 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.749226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.749404 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.749436 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:21:56.749526 (+ 90us) client_negotiation.cc:240] Negotiation successful
0809 05:21:56.749658 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13904,"thread_start_us":14339,"threads_started":1}
I20260809 05:21:56.750023 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.742770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.22.1.124:42515)
0809 05:21:56.743480 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:56.743486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:56.743501 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:21:56.743554 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:56.743559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:56.743618 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:56.743707 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:56.744327 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.745669 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.747957 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.749003 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.749130 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.749273 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:56.749590 (+ 317us) server_negotiation.cc:299] Negotiation successful
0809 05:21:56.749717 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":376,"threads_started":1}
I20260809 05:21:56.750219 32629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19299c1b7af24eb4af96c113298b06e5" is_pre_election: true
I20260809 05:21:56.750573 32629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 236e18c47d8e4d8aa3e193c7f7f4f452 in term 1.
I20260809 05:21:56.751127 407 leader_election.cc:302] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19299c1b7af24eb4af96c113298b06e5, 236e18c47d8e4d8aa3e193c7f7f4f452; no voters: b5ad151ceb04438cb667b7907a660c9a
I20260809 05:21:56.751545 1321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260809 05:21:56.751677 1321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b5ad151ceb04438cb667b7907a660c9a)
I20260809 05:21:56.751784 1321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:21:56.751981 449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "19299c1b7af24eb4af96c113298b06e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" is_pre_election: true
I20260809 05:21:56.754074 1321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } }
I20260809 05:21:56.754637 449 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 19299c1b7af24eb4af96c113298b06e5 in current term 2: Already voted for candidate 236e18c47d8e4d8aa3e193c7f7f4f452 in this term.
I20260809 05:21:56.754756 1321 leader_election.cc:288] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 election: Requested vote from peers b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321), 19299c1b7af24eb4af96c113298b06e5 (127.22.1.125:42903)
I20260809 05:21:56.755236 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.754896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43874)
0809 05:21:56.755048 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.755121 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 05:21:56.755461 32629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "19299c1b7af24eb4af96c113298b06e5"
I20260809 05:21:56.755714 32629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:21:56.755909 32589 leader_election.cc:302] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19299c1b7af24eb4af96c113298b06e5; no voters: 236e18c47d8e4d8aa3e193c7f7f4f452, b5ad151ceb04438cb667b7907a660c9a
I20260809 05:21:56.756706 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.756280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43876)
0809 05:21:56.756471 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.756575 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260809 05:21:56.756865 411 leader_election.cc:334] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321): Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
I20260809 05:21:56.758074 32629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 236e18c47d8e4d8aa3e193c7f7f4f452 in term 2.
I20260809 05:21:56.758244 1323 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:21:56.758479 407 leader_election.cc:302] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 19299c1b7af24eb4af96c113298b06e5, 236e18c47d8e4d8aa3e193c7f7f4f452; no voters: b5ad151ceb04438cb667b7907a660c9a
I20260809 05:21:56.758759 1321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 05:21:56.759202 1321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [term 2 LEADER]: Becoming Leader. State: Replica: 236e18c47d8e4d8aa3e193c7f7f4f452, State: Running, Role: LEADER
I20260809 05:21:56.759555 1321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } }
I20260809 05:21:56.761364 1336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236e18c47d8e4d8aa3e193c7f7f4f452. Latest consensus state: current_term: 2 leader_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:56.761698 1336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:56.762296 1338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:56.766355 1338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:56.767198 1338 catalog_manager.cc:1259] Loaded cluster ID: 82aabcafccfd4eada520395c3df12c1c
I20260809 05:21:56.767280 1338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:56.768391 1338 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:56.769199 1338 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Loaded TSK: 0
I20260809 05:21:56.770097 1338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:21:56.829260 432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:21:56.836580 432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:21:56.840685 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.840277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43910)
0809 05:21:56.840436 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.840535 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260809 05:21:56.840867 32629 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [term 2 FOLLOWER]: Refusing update from remote peer 236e18c47d8e4d8aa3e193c7f7f4f452: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260809 05:21:56.841599 1336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260809 05:21:56.841784 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.841323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43924)
0809 05:21:56.841508 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.841599 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 05:21:56.842003 411 consensus_peers.cc:487] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 -> Peer b5ad151ceb04438cb667b7907a660c9a (127.22.1.126:41321): Couldn't send request to peer b5ad151ceb04438cb667b7907a660c9a. Status: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260809 05:21:56.844360 1323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 236e18c47d8e4d8aa3e193c7f7f4f452. Latest consensus state: current_term: 2 leader_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:56.844650 1323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:56.846632 1321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:56.846968 1321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:56.847545 1323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b5ad151ceb04438cb667b7907a660c9a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41321 } } peers { permanent_uuid: "19299c1b7af24eb4af96c113298b06e5" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attrs { promote: false } } peers { permanent_uuid: "236e18c47d8e4d8aa3e193c7f7f4f452" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 42515 } attrs { promote: false } } }
I20260809 05:21:56.847791 1323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19299c1b7af24eb4af96c113298b06e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:21:56.860368 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.854570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.22.1.66:43763)
0809 05:21:56.855181 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:56.855188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:56.855207 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:21:56.855287 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:56.855292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:56.855364 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:56.855464 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:56.855834 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.856848 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.858664 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.859469 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.859584 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.859692 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:56.859948 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 05:21:56.860089 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":392,"threads_started":1}
I20260809 05:21:56.860353 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.854403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:43763 (local address 127.0.0.1:57988)
0809 05:21:56.854549 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.854568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:56.854676 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:56.855479 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:56.855484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:56.855507 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:56.855686 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.855698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.857002 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.857007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.858486 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.858502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.859618 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.859625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.859804 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.859827 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:21:56.859908 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:21:56.860038 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260809 05:21:56.861433 852 tablet_service.cc:1467] Processing CreateTablet for tablet 79bc104d6b36484c89bcadfa4a231117 (DEFAULT_TABLE table=test-table [id=0ee0970c50234a84a9e661ccd2452eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:56.861984 852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bc104d6b36484c89bcadfa4a231117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:56.862483 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.854727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40243 (local address 127.0.0.1:37042)
0809 05:21:56.854995 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.855022 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:56.855119 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:56.855852 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:56.855857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:56.855886 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:56.856236 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.856252 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.858136 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.858142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.860316 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.860332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.861944 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.861949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.862097 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.862130 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 05:21:56.862201 (+ 71us) client_negotiation.cc:240] Negotiation successful
0809 05:21:56.862297 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260809 05:21:56.862668 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.855045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.22.1.67:40243)
0809 05:21:56.855526 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:56.855532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:56.855552 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:21:56.855624 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:56.855629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:56.855701 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:56.855839 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:56.856427 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.857576 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.860554 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.861719 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.861890 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.862006 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:56.862234 (+ 228us) server_negotiation.cc:299] Negotiation successful
0809 05:21:56.862368 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260809 05:21:56.863379 1068 tablet_service.cc:1467] Processing CreateTablet for tablet 79bc104d6b36484c89bcadfa4a231117 (DEFAULT_TABLE table=test-table [id=0ee0970c50234a84a9e661ccd2452eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:56.863775 1068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bc104d6b36484c89bcadfa4a231117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:56.867686 1351 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Bootstrap starting.
I20260809 05:21:56.869266 1353 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Bootstrap starting.
I20260809 05:21:56.869905 1351 tablet_bootstrap.cc:654] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:56.871002 1353 tablet_bootstrap.cc:654] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:56.872803 1353 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: No bootstrap required, opened a new log
I20260809 05:21:56.872931 1353 ts_tablet_manager.cc:1397] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 05:21:56.873574 1353 raft_consensus.cc:348] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.873771 1353 raft_consensus.cc:374] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:56.873859 1353 raft_consensus.cc:729] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08f2e0339164b1a8d51add7961ad3a5, State: Initialized, Role: FOLLOWER
I20260809 05:21:56.874097 1353 consensus_queue.cc:260] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.874838 1353 ts_tablet_manager.cc:1428] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260809 05:21:56.879184 1168 heartbeater.cc:503] Master 127.22.1.124:42515 was elected leader, sending a full tablet report...
I20260809 05:21:56.885018 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.875435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:49219)
0809 05:21:56.884545 (+ 9110us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.884700 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8940,"thread_start_us":8798,"threads_started":1}
I20260809 05:21:56.885663 1351 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: No bootstrap required, opened a new log
I20260809 05:21:56.885715 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.855823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40229 (local address 127.0.0.1:45932)
0809 05:21:56.856428 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.856444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:56.856554 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:56.876070 (+ 19516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:56.876078 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:56.876111 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:56.876586 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.876600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.879893 (+ 3293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.879901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.882190 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:56.882210 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.884873 (+ 2663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.884880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:56.885082 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.885122 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 05:21:56.885213 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 05:21:56.885359 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":585,"threads_started":1}
I20260809 05:21:56.885844 1351 ts_tablet_manager.cc:1397] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.013s
I20260809 05:21:56.886063 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.856278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45932 (local address 127.22.1.65:40229)
0809 05:21:56.875584 (+ 19306us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:56.875593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:56.875617 (+ 24us) server_negotiation.cc:407] Connection header received
0809 05:21:56.875702 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:56.875708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:56.875794 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:56.877145 (+ 1351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:56.878546 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.879716 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.882437 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:56.883775 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:56.885503 (+ 1728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:56.885606 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:56.885662 (+ 56us) server_negotiation.cc:299] Negotiation successful
0809 05:21:56.885806 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19133,"thread_start_us":19611,"threads_started":1}
I20260809 05:21:56.886811 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.886439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:52531)
0809 05:21:56.886598 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.886682 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 05:21:56.886698 1351 raft_consensus.cc:348] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.886896 1351 raft_consensus.cc:374] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:56.886991 1351 raft_consensus.cc:729] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4efbab91644fa48b2e95d8054b790e, State: Initialized, Role: FOLLOWER
I20260809 05:21:56.887008 610 tablet_service.cc:1467] Processing CreateTablet for tablet 79bc104d6b36484c89bcadfa4a231117 (DEFAULT_TABLE table=test-table [id=0ee0970c50234a84a9e661ccd2452eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:21:56.887236 1351 consensus_queue.cc:260] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.887416 610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79bc104d6b36484c89bcadfa4a231117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:21:56.887979 1351 ts_tablet_manager.cc:1428] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 05:21:56.889978 942 heartbeater.cc:503] Master 127.22.1.124:42515 was elected leader, sending a full tablet report...
I20260809 05:21:56.894326 1366 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Bootstrap starting.
I20260809 05:21:56.895028 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.890595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:41403)
0809 05:21:56.894694 (+ 4099us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.894810 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3980,"thread_start_us":3853,"threads_started":1}
I20260809 05:21:56.896024 1366 tablet_bootstrap.cc:654] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:56.897938 1366 tablet_bootstrap.cc:492] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: No bootstrap required, opened a new log
I20260809 05:21:56.898065 1366 ts_tablet_manager.cc:1397] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260809 05:21:56.898691 1366 raft_consensus.cc:348] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.898869 1366 raft_consensus.cc:374] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:56.898952 1366 raft_consensus.cc:729] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97539a6b121c4f1aa9cef992d0948676, State: Initialized, Role: FOLLOWER
I20260809 05:21:56.899159 1366 consensus_queue.cc:260] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:56.899842 1366 ts_tablet_manager.cc:1428] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:21:56.901118 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.900155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:41459)
0809 05:21:56.900699 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.900857 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":293,"threads_started":1}
I20260809 05:21:56.901937 719 heartbeater.cc:503] Master 127.22.1.124:42515 was elected leader, sending a full tablet report...
I20260809 05:21:56.902130 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:56.901766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:45495)
0809 05:21:56.901902 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:56.901992 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260809 05:21:57.011885 1361 raft_consensus.cc:482] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:21:57.012135 1361 raft_consensus.cc:504] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:57.013782 1361 leader_election.cc:288] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243), 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:57.019518 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.014190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:48269 (local address 127.22.1.65:40229)
0809 05:21:57.014509 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.014541 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.014563 (+ 22us) server_negotiation.cc:407] Connection header received
0809 05:21:57.014730 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.014734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.014795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.014870 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:57.015396 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.016116 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.017840 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.018609 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.018723 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.018819 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.019121 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.019249 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260809 05:21:57.019518 1363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.014212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40229 (local address 127.22.1.66:48269)
0809 05:21:57.014427 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.014446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:57.014587 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:57.014922 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:57.014927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:57.014957 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:57.015252 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:57.015262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.016248 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.016252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:57.017686 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:57.017699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.018808 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.018815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:57.018952 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.018975 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:21:57.019071 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 05:21:57.019211 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260809 05:21:57.020030 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.014205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:42933 (local address 127.22.1.67:40243)
0809 05:21:57.014463 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.014469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.015165 (+ 696us) server_negotiation.cc:407] Connection header received
0809 05:21:57.015352 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.015356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.015414 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.015494 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:21:57.015984 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.016680 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.018346 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.019108 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.019224 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.019318 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.019651 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.019775 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:21:57.020058 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.014339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40243 (local address 127.22.1.66:42933)
0809 05:21:57.015048 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.015066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:21:57.015212 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:21:57.015519 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:21:57.015524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:21:57.015546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:21:57.015823 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:57.015832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.016808 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.016813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:57.018183 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:21:57.018197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.019323 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.019329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:21:57.019494 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.019520 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 05:21:57.019598 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 05:21:57.019718 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":399,"threads_started":1}
I20260809 05:21:57.020547 644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bc104d6b36484c89bcadfa4a231117" candidate_uuid: "2e4efbab91644fa48b2e95d8054b790e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97539a6b121c4f1aa9cef992d0948676" is_pre_election: true
I20260809 05:21:57.020839 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bc104d6b36484c89bcadfa4a231117" candidate_uuid: "2e4efbab91644fa48b2e95d8054b790e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08f2e0339164b1a8d51add7961ad3a5" is_pre_election: true
I20260809 05:21:57.020960 644 raft_consensus.cc:2454] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4efbab91644fa48b2e95d8054b790e in term 0.
I20260809 05:21:57.021113 1095 raft_consensus.cc:2454] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e4efbab91644fa48b2e95d8054b790e in term 0.
I20260809 05:21:57.021561 775 leader_election.cc:302] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4efbab91644fa48b2e95d8054b790e, 97539a6b121c4f1aa9cef992d0948676; no voters:
I20260809 05:21:57.021952 1361 raft_consensus.cc:2781] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:21:57.022097 1361 raft_consensus.cc:482] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:21:57.022198 1361 raft_consensus.cc:3037] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:57.024289 1361 raft_consensus.cc:504] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:57.024888 1361 leader_election.cc:288] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [CANDIDATE]: Term 1 election: Requested vote from peers d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243), 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229)
I20260809 05:21:57.025345 1095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bc104d6b36484c89bcadfa4a231117" candidate_uuid: "2e4efbab91644fa48b2e95d8054b790e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08f2e0339164b1a8d51add7961ad3a5"
I20260809 05:21:57.025461 644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79bc104d6b36484c89bcadfa4a231117" candidate_uuid: "2e4efbab91644fa48b2e95d8054b790e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97539a6b121c4f1aa9cef992d0948676"
I20260809 05:21:57.025599 1095 raft_consensus.cc:3037] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:57.025655 644 raft_consensus.cc:3037] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:57.028579 1095 raft_consensus.cc:2454] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4efbab91644fa48b2e95d8054b790e in term 1.
I20260809 05:21:57.028602 644 raft_consensus.cc:2454] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e4efbab91644fa48b2e95d8054b790e in term 1.
I20260809 05:21:57.029052 769 leader_election.cc:302] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4efbab91644fa48b2e95d8054b790e, d08f2e0339164b1a8d51add7961ad3a5; no voters:
I20260809 05:21:57.029409 1361 raft_consensus.cc:2781] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:57.029978 1361 raft_consensus.cc:686] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 1 LEADER]: Becoming Leader. State: Replica: 2e4efbab91644fa48b2e95d8054b790e, State: Running, Role: LEADER
I20260809 05:21:57.030412 1361 consensus_queue.cc:237] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } }
I20260809 05:21:57.030994 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.030581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:38827)
0809 05:21:57.030743 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.030845 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260809 05:21:57.034260 432 catalog_manager.cc:5495] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66). New cstate: current_term: 1 leader_uuid: "2e4efbab91644fa48b2e95d8054b790e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4efbab91644fa48b2e95d8054b790e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260809 05:21:57.055462 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.050705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.22.1.66:43763)
0809 05:21:57.050881 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.050886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.050902 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:21:57.050989 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.050993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.051078 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.051200 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:21:57.051468 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.052165 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.053422 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.053837 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.053930 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.054373 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:21:57.054905 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:21:57.055029 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.055130 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.055201 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:21:57.059374 1095 raft_consensus.cc:1261] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Refusing update from remote peer 2e4efbab91644fa48b2e95d8054b790e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:57.059665 644 raft_consensus.cc:1261] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Refusing update from remote peer 2e4efbab91644fa48b2e95d8054b790e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:21:57.060072 1372 consensus_queue.cc:1035] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08f2e0339164b1a8d51add7961ad3a5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:57.060449 1361 consensus_queue.cc:1035] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [LEADER]: Connected to new peer: Peer: permanent_uuid: "97539a6b121c4f1aa9cef992d0948676" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:21:57.063248 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.062529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:39527)
0809 05:21:57.062932 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.063055 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294}
I20260809 05:21:57.064218 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.063659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:36543)
0809 05:21:57.063892 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.064000 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260809 05:21:57.069366 1377 mvcc.cc:205] Tried to move back new op lower bound from 7316491948268634112 to 7316491948160999424. Current Snapshot: MvccSnapshot[applied={T|T < 7316491948268634112}]
I20260809 05:21:57.070577 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.070075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.67:49947)
0809 05:21:57.070326 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.070411 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 05:21:57.071506 1376 mvcc.cc:205] Tried to move back new op lower bound from 7316491948268634112 to 7316491948160999424. Current Snapshot: MvccSnapshot[applied={T|T < 7316491948268634112 or (T in {7316491948268634112})}]
I20260809 05:21:57.072108 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.071622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:38819)
0809 05:21:57.071805 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.071909 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 05:21:57.081185 1371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.080542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.66:33263)
0809 05:21:57.080842 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.080963 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260809 05:21:57.085708 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.085131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.22.1.65:42215)
0809 05:21:57.085448 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.085546 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20260809 05:21:57.085533 431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:57.086272 431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:21:57.092844 431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 236e18c47d8e4d8aa3e193c7f7f4f452: Sending DeleteTablet for 3 replicas of tablet 79bc104d6b36484c89bcadfa4a231117
I20260809 05:21:57.093798 852 tablet_service.cc:1514] Processing DeleteTablet for tablet 79bc104d6b36484c89bcadfa4a231117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:57 UTC) from {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:57988
I20260809 05:21:57.093858 1068 tablet_service.cc:1514] Processing DeleteTablet for tablet 79bc104d6b36484c89bcadfa4a231117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:57 UTC) from {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:37042
I20260809 05:21:57.095062 1390 tablet_replica.cc:330] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: stopping tablet replica
I20260809 05:21:57.095338 1390 raft_consensus.cc:2229] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:21:57.096058 1390 raft_consensus.cc:2258] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:57.096513 610 tablet_service.cc:1514] Processing DeleteTablet for tablet 79bc104d6b36484c89bcadfa4a231117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:57 UTC) from {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:45932
I20260809 05:21:57.097007 608 tablet_service.cc:1514] Processing DeleteTablet for tablet 79bc104d6b36484c89bcadfa4a231117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:21:57 UTC) from {username='oryx', principal='oryx/127.22.1.124@KRBTEST.COM'} at 127.0.0.1:45932
I20260809 05:21:57.097545 1390 ts_tablet_manager.cc:1905] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:57.099267 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 393
I20260809 05:21:57.103533 1392 tablet_replica.cc:330] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: stopping tablet replica
I20260809 05:21:57.103785 1392 raft_consensus.cc:2229] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:57.104156 1392 raft_consensus.cc:2258] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:57.107074 1390 ts_tablet_manager.cc:1918] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:57.107247 1390 log.cc:1196] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/79bc104d6b36484c89bcadfa4a231117
I20260809 05:21:57.107757 1390 ts_tablet_manager.cc:1939] T 79bc104d6b36484c89bcadfa4a231117 P 2e4efbab91644fa48b2e95d8054b790e: Deleting consensus metadata
I20260809 05:21:57.109128 412 catalog_manager.cc:4841] TS 2e4efbab91644fa48b2e95d8054b790e (127.22.1.66:43763): tablet 79bc104d6b36484c89bcadfa4a231117 (table test-table [id=0ee0970c50234a84a9e661ccd2452eb8]) successfully deleted
I20260809 05:21:57.109800 1394 tablet_replica.cc:330] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: stopping tablet replica
I20260809 05:21:57.110015 1394 raft_consensus.cc:2229] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:21:57.110287 1392 ts_tablet_manager.cc:1905] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:57.110339 1394 raft_consensus.cc:2258] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:21:57.111243 1394 ts_tablet_manager.cc:1905] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:21:57.116641 1394 ts_tablet_manager.cc:1918] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:57.116787 1394 log.cc:1196] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/79bc104d6b36484c89bcadfa4a231117
I20260809 05:21:57.117292 1394 ts_tablet_manager.cc:1939] T 79bc104d6b36484c89bcadfa4a231117 P 97539a6b121c4f1aa9cef992d0948676: Deleting consensus metadata
I20260809 05:21:57.117554 1392 ts_tablet_manager.cc:1918] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:21:57.117686 1392 log.cc:1196] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/79bc104d6b36484c89bcadfa4a231117
I20260809 05:21:57.118132 1392 ts_tablet_manager.cc:1939] T 79bc104d6b36484c89bcadfa4a231117 P d08f2e0339164b1a8d51add7961ad3a5: Deleting consensus metadata
I20260809 05:21:57.118476 412 catalog_manager.cc:4841] TS 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229): tablet 79bc104d6b36484c89bcadfa4a231117 (table test-table [id=0ee0970c50234a84a9e661ccd2452eb8]) successfully deleted
I20260809 05:21:57.119241 405 catalog_manager.cc:4841] TS d08f2e0339164b1a8d51add7961ad3a5 (127.22.1.67:40243): tablet 79bc104d6b36484c89bcadfa4a231117 (table test-table [id=0ee0970c50234a84a9e661ccd2452eb8]) successfully deleted
W20260809 05:21:57.119532 412 catalog_manager.cc:4808] TS 97539a6b121c4f1aa9cef992d0948676 (127.22.1.65:40229): delete failed for tablet 79bc104d6b36484c89bcadfa4a231117 because the tablet was not found. No further retry: Not found: Tablet not found: 79bc104d6b36484c89bcadfa4a231117
I20260809 05:21:57.128827 1349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.100665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.22.1.65:40229)
0809 05:21:57.100862 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.100868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.114745 (+ 13877us) server_negotiation.cc:407] Connection header received
0809 05:21:57.114959 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.114965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.115054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.115127 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:57.117256 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.118288 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.120181 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.120572 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.120684 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.123179 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:57.123190 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:57.123207 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:57.123238 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:57.126121 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.126793 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.126801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.126803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.126930 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.127298 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.127302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.127304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.127654 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:57.127987 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.128266 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.128497 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:21:57.129559 714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:57.253723 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 393
I20260809 05:21:57.283773 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 746
I20260809 05:21:57.299440 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.284593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.22.1.66:43763)
0809 05:21:57.284781 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.284788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.284809 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:21:57.284984 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.284989 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.285067 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.285202 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:57.286486 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.287664 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.288881 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.289657 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.289789 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.292997 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:57.293020 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:57.293036 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:57.293072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:57.296313 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.297223 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.297231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.297234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.297382 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.297820 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.297824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.297826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.298214 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:57.298513 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.298912 (+ 399us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.299114 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:21:57.300128 935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:57.416498 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 746
I20260809 05:21:57.441636 1405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.440763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43932)
0809 05:21:57.441305 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.441424 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":465,"threads_started":1}
I20260809 05:21:57.441725 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 961
I20260809 05:21:57.443310 1405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.442912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41321 (local address 127.0.0.1:43946)
0809 05:21:57.443074 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0809 05:21:57.443163 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 05:21:57.454921 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:57.442509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.22.1.67:40243)
0809 05:21:57.442680 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:57.442684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:57.442702 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:21:57.442850 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:57.442856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:57.442941 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:57.443055 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:57.444536 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.445456 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.446623 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:57.447080 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:57.447214 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:57.449717 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:57.449733 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:57.449752 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:57.449787 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:57.452527 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.453139 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.453146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.453177 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.453314 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:57.453624 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:57.453626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:57.453629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:57.454008 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:57.454243 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:57.454432 (+ 189us) server_negotiation.cc:299] Negotiation successful
0809 05:21:57.454619 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:21:57.455606 1158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:57.547557 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 961
I20260809 05:21:57.569199 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 32406
I20260809 05:21:57.570333 32658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:57.689108 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 32406
I20260809 05:21:57.712291 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 32553
I20260809 05:21:57.713651 478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:21:57.835443 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 32553
2026-08-09T05:21:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10592 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:21:57 dist-test-slave-rmhg krb5kdc[1412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:21:57 dist-test-slave-rmhg krb5kdc[1412](info): set up 2 sockets
Aug 09 05:21:57 dist-test-slave-rmhg krb5kdc[1412](info): commencing operation
krb5kdc: starting...
W20260809 05:21:59.039086 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:21:59 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:21:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab.
Entry for principal oryx/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab.
I20260809 05:21:59.280831 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:46805
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab
--principal=oryx/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:21:59.617175 1437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:21:59.617533 1437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:21:59.617666 1437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:21:59.625039 1437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:21:59.625201 1437 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260809 05:21:59.625283 1437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:21:59.625367 1437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:21:59.625435 1437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:21:59.625530 1437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:21:59.635820 1437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:46805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.126.keytab
--principal=oryx/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:21:59.637276 1437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:21:59.638692 1437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:21:59.650720 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:59.650679 1449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:21:59.651098 1445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:21:59.651461 1437 server_base.cc:1034] running on GCE node
I20260809 05:21:59.652621 1437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:21:59.654372 1437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:21:59.655608 1437 hybrid_clock.cc:648] HybridClock initialized: now 1786252919655547 us; error 35 us; skew 500 ppm
I20260809 05:21:59.655920 1437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:21:59 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252919, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:21:59.659133 1437 init.cc:377] Logged in from keytab as oryx/127.22.1.126@KRBTEST.COM (short username oryx)
I20260809 05:21:59.660714 1437 webserver.cc:460] Webserver started at http://127.22.1.126:43401/ using document root <none> and password file <none>
I20260809 05:21:59.661489 1437 fs_manager.cc:359] Metadata directory not provided
I20260809 05:21:59.661583 1437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:21:59.661908 1437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:21:59.665043 1437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "11cc1d0bfe6a4ae08159d68af2330864"
format_stamp: "Formatted at 2026-08-09 05:21:59 on dist-test-slave-rmhg"
server_key: "6f2a775d81ea92a7ce5b473051b6452d"
server_key_iv: "c1c09384a137ec1b963f8e0fb1f222ed"
server_key_version: "encryptionkey@0"
I20260809 05:21:59.665899 1437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "11cc1d0bfe6a4ae08159d68af2330864"
format_stamp: "Formatted at 2026-08-09 05:21:59 on dist-test-slave-rmhg"
server_key: "6f2a775d81ea92a7ce5b473051b6452d"
server_key_iv: "c1c09384a137ec1b963f8e0fb1f222ed"
server_key_version: "encryptionkey@0"
I20260809 05:21:59.671207 1437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:21:59.674948 1460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:21:59.676389 1437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:21:59.676555 1437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "11cc1d0bfe6a4ae08159d68af2330864"
format_stamp: "Formatted at 2026-08-09 05:21:59 on dist-test-slave-rmhg"
server_key: "6f2a775d81ea92a7ce5b473051b6452d"
server_key_iv: "c1c09384a137ec1b963f8e0fb1f222ed"
server_key_version: "encryptionkey@0"
I20260809 05:21:59.676757 1437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:21:59.697144 1437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:21:59.701437 1437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:21:59.701745 1437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:21:59.723074 1437 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:46805
I20260809 05:21:59.723095 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:46805 every 8 connection(s)
I20260809 05:21:59.724467 1437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:21:59.725807 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 1437
I20260809 05:21:59.726048 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:21:59.726900 22533 external_mini_cluster.cc:1428] Setting key 45005d77abc0b88de4716d1a7b9c6f07
I20260809 05:21:59.729188 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 05:21:59 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:21:59.742708 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Bootstrap starting.
I20260809 05:21:59.746182 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Neither blocks nor log segments found. Creating new log.
I20260809 05:21:59.747447 1559 log.cc:824] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Log is configured to *not* fsync() on all Append() calls
I20260809 05:21:59.750321 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: No bootstrap required, opened a new log
I20260809 05:21:59.750511 1563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:21:59.729273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.22.1.126:46805)
0809 05:21:59.730114 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0809 05:21:59.730127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:21:59.730169 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:21:59.731836 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:21:59.731860 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:21:59.732275 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:21:59.732615 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:21:59.734446 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:59.735837 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:59.737517 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:21:59.738194 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:21:59.738318 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:21:59.743075 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:21:59.743084 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:21:59.743100 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:21:59.743137 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:21:59.746184 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:59.747064 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:59.747071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:59.747078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:59.747252 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:21:59.747696 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:21:59.747700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:21:59.747703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:21:59.748257 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:21:59.748557 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 05:21:59.749026 (+ 469us) server_negotiation.cc:299] Negotiation successful
0809 05:21:59.749504 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":354,"threads_started":1}
I20260809 05:21:59.758401 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } }
I20260809 05:21:59.758802 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:21:59.758895 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11cc1d0bfe6a4ae08159d68af2330864, State: Initialized, Role: FOLLOWER
I20260809 05:21:59.759502 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } }
I20260809 05:21:59.759755 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:21:59.759943 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:21:59.760118 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:21:59.762326 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } }
I20260809 05:21:59.762816 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11cc1d0bfe6a4ae08159d68af2330864; no voters:
I20260809 05:21:59.763383 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:21:59.763500 1567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:21:59.764150 1567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [term 1 LEADER]: Becoming Leader. State: Replica: 11cc1d0bfe6a4ae08159d68af2330864, State: Running, Role: LEADER
I20260809 05:21:59.764801 1567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } }
I20260809 05:21:59.765666 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:21:59.767822 1571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11cc1d0bfe6a4ae08159d68af2330864. Latest consensus state: current_term: 1 leader_uuid: "11cc1d0bfe6a4ae08159d68af2330864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } } }
I20260809 05:21:59.768258 1571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:59.767944 1568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11cc1d0bfe6a4ae08159d68af2330864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11cc1d0bfe6a4ae08159d68af2330864" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 46805 } } }
I20260809 05:21:59.768590 1568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864 [sys.catalog]: This master's current role is: LEADER
I20260809 05:21:59.770756 1582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:21:59.775876 1582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:21:59.785449 1582 catalog_manager.cc:1347] Generated new cluster ID: 2a352762e2434e95ab2e4e146d8b98d7
I20260809 05:21:59.785565 1582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:21:59.795533 1582 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:21:59.796898 1582 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:21:59.809005 1582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Generated new TSK 0
I20260809 05:21:59.809930 1582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
I20260809 05:21:59.878176 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46805
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:00.215329 1600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:00.215626 1600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:00.215765 1600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:00.223515 1600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:00.223645 1600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:00.223775 1600 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:22:00.223881 1600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:00.233912 1600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:00.235260 1600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:00.236634 1600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:00.249943 1608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:00.250057 1611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:00.250524 1607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:00.250810 1600 server_base.cc:1034] running on GCE node
I20260809 05:22:00.251483 1600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:00.252399 1600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:00.253620 1600 hybrid_clock.cc:648] HybridClock initialized: now 1786252920253542 us; error 46 us; skew 500 ppm
I20260809 05:22:00.253914 1600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:00 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252920, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:00.256925 1600 init.cc:377] Logged in from keytab as oryx/127.22.1.65@KRBTEST.COM (short username oryx)
I20260809 05:22:00.258386 1600 webserver.cc:460] Webserver started at http://127.22.1.65:35965/ using document root <none> and password file <none>
I20260809 05:22:00.259085 1600 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:00.259189 1600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:00.259510 1600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:00.262514 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "33f4f98f54ce453eb523f6384cb40062"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "f0fb166abf61e744ace0bd992d74589f"
server_key_iv: "246a1048c9d50954195d03bcdfe392c9"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.263335 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "33f4f98f54ce453eb523f6384cb40062"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "f0fb166abf61e744ace0bd992d74589f"
server_key_iv: "246a1048c9d50954195d03bcdfe392c9"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.268639 1600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:22:00.272181 1622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:00.273555 1600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260809 05:22:00.273705 1600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "33f4f98f54ce453eb523f6384cb40062"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "f0fb166abf61e744ace0bd992d74589f"
server_key_iv: "246a1048c9d50954195d03bcdfe392c9"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.273912 1600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:00.286772 1600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:00.290649 1600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:00.290896 1600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:00.293663 1600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:00.293809 1600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260809 05:22:00.293943 1600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:00.294014 1600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:22:00 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252920, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:00.317713 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.294369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:41063 (local address 127.22.1.126:46805)
0809 05:22:00.294906 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:00.294910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:00.296132 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 05:22:00.298303 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:00.298309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:00.298398 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:00.298505 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:00.300795 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.301878 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.304721 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.305441 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.305576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.310294 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:00.310300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:00.310303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:00.310338 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:00.313609 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.314656 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.314662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.314666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.314814 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.315388 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.315393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.315397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.315783 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:00.315926 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:00.317042 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0809 05:22:00.317325 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":334,"threads_started":1}
I20260809 05:22:00.318894 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.295099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.65:41063)
0809 05:22:00.295967 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:00.296010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:00.298019 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:00.298791 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:00.298802 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:00.299660 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:00.300542 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.300573 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.302129 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.302134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.304541 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.304559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.305691 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.305696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.305881 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.306611 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:00.306650 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:00.310036 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:00.313789 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.313797 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.313816 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.314473 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.314942 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.314947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.314954 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.315278 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.316017 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:00.316028 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:00.316550 (+ 522us) client_negotiation.cc:769] Sending connection context
0809 05:22:00.316960 (+ 410us) client_negotiation.cc:240] Negotiation successful
0809 05:22:00.317617 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":379,"threads_started":1}
I20260809 05:22:00.343309 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.325413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:46255 (local address 127.22.1.126:46805)
0809 05:22:00.325627 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:00.325634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:00.329828 (+ 4194us) server_negotiation.cc:407] Connection header received
0809 05:22:00.330020 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:00.330026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:00.330105 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:00.330191 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:00.331206 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.331921 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.333295 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.333859 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.333977 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.336345 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:00.336352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:00.336358 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:00.336405 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:00.339804 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.340892 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.340898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.340902 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.341105 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.341609 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.341615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.341618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.342050 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:00.342218 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:00.342706 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 05:22:00.342932 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:22:00.343461 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.325350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.65:46255)
0809 05:22:00.329686 (+ 4336us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:00.329709 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:00.329887 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:00.330238 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:00.330243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:00.330730 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:00.331043 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.331054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.332064 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.332068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.333095 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.333108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.334085 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.334092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.334259 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.335040 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:00.335065 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:00.336123 (+ 1058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:00.339975 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.339982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.339986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.340735 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.341244 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.341250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.341254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.341445 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.342189 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:00.342193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:00.342375 (+ 182us) client_negotiation.cc:769] Sending connection context
0809 05:22:00.342561 (+ 186us) client_negotiation.cc:240] Negotiation successful
0809 05:22:00.342976 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4176,"thread_start_us":4032,"threads_started":1}
I20260809 05:22:00.345952 1600 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:37757
I20260809 05:22:00.345952 1818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:37757 every 8 connection(s)
I20260809 05:22:00.347240 1600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:00.349256 1820 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46805
I20260809 05:22:00.349448 1820 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:00.350054 1820 heartbeater.cc:511] Master 127.22.1.126:46805 requested a full tablet report, sending...
I20260809 05:22:00.351967 1487 ts_manager.cc:194] Registered new tserver with Master: 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757)
I20260809 05:22:00.353535 1487 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:41063
I20260809 05:22:00.355147 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 1600
I20260809 05:22:00.355340 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:22:00.355960 22533 external_mini_cluster.cc:1428] Setting key dad13c40954bcd6e86ca97b3075e72b5
WARNING: no policy specified for oryx/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
I20260809 05:22:00.407913 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46805
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:00.741705 1828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:00.742014 1828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:00.742147 1828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:00.749428 1828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:00.749564 1828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:00.749694 1828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:22:00.749807 1828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:00.759716 1828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:00.761116 1828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:00.762542 1828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:00.775794 1837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:00.775817 1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:00.776307 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:00.776573 1828 server_base.cc:1034] running on GCE node
I20260809 05:22:00.777320 1828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:00.778330 1828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:00.779546 1828 hybrid_clock.cc:648] HybridClock initialized: now 1786252920779478 us; error 46 us; skew 500 ppm
I20260809 05:22:00.779862 1828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:00 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252920, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:00.783159 1828 init.cc:377] Logged in from keytab as oryx/127.22.1.66@KRBTEST.COM (short username oryx)
I20260809 05:22:00.784682 1828 webserver.cc:460] Webserver started at http://127.22.1.66:33965/ using document root <none> and password file <none>
I20260809 05:22:00.785485 1828 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:00.785602 1828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:00.785920 1828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:00.788884 1828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "09a0db971ea4466eabbc49796a93cfe3"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "21b0a8d33fe479ae9e745afc725583ce"
server_key_iv: "8bf9bf8efba61b1ad024712038c03742"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.789755 1828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "09a0db971ea4466eabbc49796a93cfe3"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "21b0a8d33fe479ae9e745afc725583ce"
server_key_iv: "8bf9bf8efba61b1ad024712038c03742"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.795086 1828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:22:00.798825 1853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:00.800184 1828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:22:00.800346 1828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "09a0db971ea4466eabbc49796a93cfe3"
format_stamp: "Formatted at 2026-08-09 05:22:00 on dist-test-slave-rmhg"
server_key: "21b0a8d33fe479ae9e745afc725583ce"
server_key_iv: "8bf9bf8efba61b1ad024712038c03742"
server_key_version: "encryptionkey@0"
I20260809 05:22:00.800561 1828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:00.819550 1828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:00.823824 1828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:00.824061 1828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:00.826819 1828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:00.826927 1828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:00.827064 1828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:00.827135 1828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:22:00 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252920, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:00.850634 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.827820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49659 (local address 127.22.1.126:46805)
0809 05:22:00.827995 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:00.828000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:00.829327 (+ 1327us) server_negotiation.cc:407] Connection header received
0809 05:22:00.831520 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:00.831526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:00.831614 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:00.831714 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:00.834410 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.835346 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.837583 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.838257 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.838390 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.843059 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:00.843064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:00.843068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:00.843106 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:00.846228 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.847270 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.847276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.847279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.847423 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.848120 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.848127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.848131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.848527 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:00.848705 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:00.849966 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0809 05:22:00.850256 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 05:22:00.851729 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.828254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.66:49659)
0809 05:22:00.829114 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:00.829201 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:00.831211 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:00.832150 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:00.832163 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:00.833141 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:00.834184 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.834214 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.835521 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.835528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.837388 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.837405 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.838547 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.838553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.838751 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.839594 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:00.839618 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:00.842794 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:00.846416 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.846424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.846439 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.847104 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.847551 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.847557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.847560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.847929 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.848684 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:00.848694 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:00.849125 (+ 431us) client_negotiation.cc:769] Sending connection context
0809 05:22:00.849539 (+ 414us) client_negotiation.cc:240] Negotiation successful
0809 05:22:00.850162 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":367,"threads_started":1}
I20260809 05:22:00.871634 1828 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:44113
I20260809 05:22:00.871651 2022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:44113 every 8 connection(s)
I20260809 05:22:00.873008 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.859849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:59635 (local address 127.22.1.126:46805)
0809 05:22:00.860044 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:00.860049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:00.860188 (+ 139us) server_negotiation.cc:407] Connection header received
0809 05:22:00.860408 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:00.860413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:00.860497 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:00.860592 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:00.861662 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.862456 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.863804 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.864235 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.864353 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.866845 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:00.866854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:00.866860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:00.866905 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:00.870132 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.870908 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.870914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.870918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.871079 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:00.871458 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:00.871463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:00.871467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:00.871883 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:00.872035 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:00.872453 (+ 418us) server_negotiation.cc:299] Negotiation successful
0809 05:22:00.872668 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:22:00.873054 1828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:00.873128 1994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:00.859285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.66:59635)
0809 05:22:00.860066 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:00.860088 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:00.860272 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:00.860644 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:00.860653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:00.861171 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:00.861514 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.861525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.862594 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.862599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.863663 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:00.863675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:00.864424 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:00.864448 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:00.864614 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:00.865438 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:00.865463 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:00.866630 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:00.870261 (+ 3631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.870267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.870272 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.870737 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.871194 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:00.871200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:00.871203 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:00.871339 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:00.872005 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:00.872010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:00.872198 (+ 188us) client_negotiation.cc:769] Sending connection context
0809 05:22:00.872327 (+ 129us) client_negotiation.cc:240] Negotiation successful
0809 05:22:00.872681 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":618,"threads_started":1}
I20260809 05:22:00.873373 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 1828
I20260809 05:22:00.873576 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:22:00.874173 22533 external_mini_cluster.cc:1428] Setting key 0b9a82f915ce5384b45e70d6587fa9e4
I20260809 05:22:00.875391 2023 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46805
I20260809 05:22:00.875648 2023 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:00.876658 2023 heartbeater.cc:511] Master 127.22.1.126:46805 requested a full tablet report, sending...
I20260809 05:22:00.878049 1485 ts_manager.cc:194] Registered new tserver with Master: 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113)
I20260809 05:22:00.879176 1485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49659
WARNING: no policy specified for oryx/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
I20260809 05:22:00.927873 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:46805
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:01.274345 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:01.274670 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:01.274817 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:01.282413 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:01.282552 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:01.282678 2028 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260809 05:22:01.282790 2028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:01.292907 2028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.1.126:46805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:01.294368 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:01.295819 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:01.310755 2040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:01.310811 2037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:01.310603 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:01.311654 2028 server_base.cc:1034] running on GCE node
I20260809 05:22:01.312348 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:01.313364 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:01.314646 2028 hybrid_clock.cc:648] HybridClock initialized: now 1786252921314572 us; error 46 us; skew 500 ppm
I20260809 05:22:01.314957 2028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252921, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:01.318216 2028 init.cc:377] Logged in from keytab as oryx/127.22.1.67@KRBTEST.COM (short username oryx)
I20260809 05:22:01.319703 2028 webserver.cc:460] Webserver started at http://127.22.1.67:39631/ using document root <none> and password file <none>
I20260809 05:22:01.320431 2028 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:01.320534 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:01.320871 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:01.324115 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "3943e990350a40ac833ccbc92b3eea39"
format_stamp: "Formatted at 2026-08-09 05:22:01 on dist-test-slave-rmhg"
server_key: "f0adfabf28374263fcccdc9ea3b844b3"
server_key_iv: "d368a4a8397a207a5ca085d964df8b44"
server_key_version: "encryptionkey@0"
I20260809 05:22:01.324996 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "3943e990350a40ac833ccbc92b3eea39"
format_stamp: "Formatted at 2026-08-09 05:22:01 on dist-test-slave-rmhg"
server_key: "f0adfabf28374263fcccdc9ea3b844b3"
server_key_iv: "d368a4a8397a207a5ca085d964df8b44"
server_key_version: "encryptionkey@0"
I20260809 05:22:01.330870 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 05:22:01.335166 2053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:01.336925 2028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260809 05:22:01.337138 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "3943e990350a40ac833ccbc92b3eea39"
format_stamp: "Formatted at 2026-08-09 05:22:01 on dist-test-slave-rmhg"
server_key: "f0adfabf28374263fcccdc9ea3b844b3"
server_key_iv: "d368a4a8397a207a5ca085d964df8b44"
server_key_version: "encryptionkey@0"
I20260809 05:22:01.337436 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:01.353560 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:01.356256 1820 heartbeater.cc:503] Master 127.22.1.126:46805 was elected leader, sending a full tablet report...
I20260809 05:22:01.358177 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:01.358450 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:01.361379 2028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:01.361475 2028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:01.361630 2028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:01.362030 2028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252921, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:01.384747 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.361949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:34043 (local address 127.22.1.126:46805)
0809 05:22:01.362153 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.362160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.363637 (+ 1477us) server_negotiation.cc:407] Connection header received
0809 05:22:01.365841 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.365848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.365951 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.366081 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:01.368457 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.369664 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.372129 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.372809 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.372952 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.377975 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:01.377983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:01.377987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:01.378044 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:01.381172 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.382273 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.382278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.382280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.382418 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.382900 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.382903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.382905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.383190 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:01.383317 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.384280 (+ 963us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.384486 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:22:01.386837 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.362484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.67:34043)
0809 05:22:01.363417 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.363478 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.365521 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.366437 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.366453 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.367297 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:01.368212 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.368238 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.369852 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.369858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.371955 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.371970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.373125 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.373134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.373514 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.374380 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:01.374420 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:01.377642 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:01.381336 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.381348 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.381370 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.382139 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.382518 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.382523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.382526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.382802 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.383307 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:01.383317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:01.383845 (+ 528us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.384201 (+ 356us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.384869 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":373,"threads_started":1}
I20260809 05:22:01.406921 2028 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40427
I20260809 05:22:01.406963 2235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40427 every 8 connection(s)
I20260809 05:22:01.408573 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:01.410750 2236 heartbeater.cc:347] Connected to a master server at 127.22.1.126:46805
I20260809 05:22:01.411010 2236 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:01.411883 2236 heartbeater.cc:511] Master 127.22.1.126:46805 requested a full tablet report, sending...
I20260809 05:22:01.412853 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 2028
I20260809 05:22:01.413082 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:22:01.413993 1485 ts_manager.cc:194] Registered new tserver with Master: 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427)
I20260809 05:22:01.415169 22533 external_mini_cluster.cc:1428] Setting key da87d095021d6849d6e6f6b489926e99
I20260809 05:22:01.416636 1485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:34043
I20260809 05:22:01.418962 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:22:01.419354 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.403914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:45053 (local address 127.22.1.126:46805)
0809 05:22:01.404119 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.404125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.405080 (+ 955us) server_negotiation.cc:407] Connection header received
0809 05:22:01.405331 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.405337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.405438 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.405558 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:01.406901 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.407970 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.409598 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.410151 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.410252 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.412641 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:01.412648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:01.412652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:01.412693 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:01.416189 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.417075 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.417081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.417085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.417276 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.417722 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.417728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.417731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.418088 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:01.418242 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.418743 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.418978 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 05:22:01.420466 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.403939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.22.1.67:45053)
0809 05:22:01.404928 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.404949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.405186 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.405578 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.405581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.406256 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:01.406719 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.406736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.408134 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.408138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.409404 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.409418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.410541 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.410546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.410717 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.411350 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:01.411373 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:01.412404 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:01.416369 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.416375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.416379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.416913 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.417442 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.417446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.417450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.417606 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.418227 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:01.418232 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:01.418436 (+ 204us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.419602 (+ 1166us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.419981 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":542,"threads_started":1}
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 05:22:01.476297 1637 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:01.466344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.22.1.126:46805)
0809 05:22:01.466501 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.466526 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.466779 (+ 253us) server_negotiation.cc:407] Connection header received
0809 05:22:01.466956 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.466960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.467040 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.467117 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:01.468369 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.469211 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.470526 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.470943 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.471040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.476070 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38972: BlockingRecv error: recv got EOF from 127.0.0.1:38972 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252921, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:01.491288 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.479149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.22.1.126:46805)
0809 05:22:01.479330 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.479336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.479540 (+ 204us) server_negotiation.cc:407] Connection header received
0809 05:22:01.479691 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.479694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.479789 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.479870 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:01.480918 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.481698 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.482822 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.483209 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.483292 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.486630 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:01.486635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:01.486639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:01.486672 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:01.489085 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.489752 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.489756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.489758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.489875 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.490177 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.490181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.490183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.490453 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:01.490693 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.490859 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.491037 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 05:22:01.495292 1485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:22:01.502105 1485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:01.535858 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.516952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.22.1.65:37757)
0809 05:22:01.527821 (+ 10869us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.527829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.527854 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:22:01.527940 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.527947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.528168 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.528354 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.529549 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.531069 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.533264 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.534659 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.534801 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.535025 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.535369 (+ 344us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.535563 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10717,"thread_start_us":10570,"threads_started":1}
I20260809 05:22:01.535975 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.516684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.0.0.1:55860)
0809 05:22:01.527027 (+ 10343us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.527046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.527195 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.528527 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.528532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.528559 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.529081 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.529101 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.531264 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.531274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.533038 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.533056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.534875 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.534881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.535096 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.535158 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.535300 (+ 142us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.535505 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10234,"mutex_wait_us":1,"thread_start_us":9453,"threads_started":1}
I20260809 05:22:01.536083 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.514602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:44113 (local address 127.0.0.1:37586)
0809 05:22:01.526452 (+ 11850us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.526475 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.527340 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.528273 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.528277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.528295 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.528996 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.529015 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.530733 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.530738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.533206 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.533222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.535202 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.535210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.535371 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.535404 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.535493 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.535641 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11700,"mutex_wait_us":76,"thread_start_us":12186,"threads_started":1}
I20260809 05:22:01.536689 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.515098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37586 (local address 127.22.1.66:44113)
0809 05:22:01.525699 (+ 10601us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.525710 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.526614 (+ 904us) server_negotiation.cc:407] Connection header received
0809 05:22:01.527567 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.527577 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.527878 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.528038 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.529279 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.530610 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.533882 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.534833 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.534966 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.535167 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.536187 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.536375 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10378,"thread_start_us":10619,"threads_started":1}
I20260809 05:22:01.537045 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.514394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40427 (local address 127.0.0.1:37244)
0809 05:22:01.515508 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.515563 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.515752 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.528231 (+ 12479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.528239 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.528274 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.528996 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.529021 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.531216 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.531222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.533622 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.533635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.535765 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.535772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.536442 (+ 670us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.536480 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.536568 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.536697 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":104,"thread_start_us":900,"threads_started":1}
I20260809 05:22:01.537531 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.514636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.22.1.67:40427)
0809 05:22:01.527517 (+ 12881us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.527527 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.527565 (+ 38us) server_negotiation.cc:407] Connection header received
0809 05:22:01.527674 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.527680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.527940 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.528175 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.529331 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.531065 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.533802 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.535268 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.535391 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.535609 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.536987 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.537204 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12699,"thread_start_us":12514,"threads_started":1}
I20260809 05:22:01.540177 1932 tablet_service.cc:1467] Processing CreateTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 (DEFAULT_TABLE table=test-table [id=315f53d7e526402da08b5ff157a1e78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.540509 1705 tablet_service.cc:1467] Processing CreateTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 (DEFAULT_TABLE table=test-table [id=315f53d7e526402da08b5ff157a1e78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.540839 2129 tablet_service.cc:1467] Processing CreateTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 (DEFAULT_TABLE table=test-table [id=315f53d7e526402da08b5ff157a1e78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.541622 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c86bd130464501b09bd776ed8cbcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.542035 1705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c86bd130464501b09bd776ed8cbcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.542297 2129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c86bd130464501b09bd776ed8cbcf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.553323 2276 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Bootstrap starting.
I20260809 05:22:01.556550 2277 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Bootstrap starting.
I20260809 05:22:01.558513 2278 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Bootstrap starting.
I20260809 05:22:01.558945 2276 tablet_bootstrap.cc:654] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.560124 2277 tablet_bootstrap.cc:654] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.560389 2276 log.cc:824] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:01.561393 2277 log.cc:824] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:01.561528 2278 tablet_bootstrap.cc:654] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.562500 2278 log.cc:824] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:01.563813 2276 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: No bootstrap required, opened a new log
I20260809 05:22:01.564082 2276 ts_tablet_manager.cc:1397] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260809 05:22:01.565222 2277 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: No bootstrap required, opened a new log
I20260809 05:22:01.565454 2278 tablet_bootstrap.cc:492] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: No bootstrap required, opened a new log
I20260809 05:22:01.565492 2277 ts_tablet_manager.cc:1397] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260809 05:22:01.565739 2278 ts_tablet_manager.cc:1397] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260809 05:22:01.572331 2276 raft_consensus.cc:348] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.572726 2276 raft_consensus.cc:374] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.572819 2276 raft_consensus.cc:729] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.573487 2276 consensus_queue.cc:260] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.574616 2023 heartbeater.cc:503] Master 127.22.1.126:46805 was elected leader, sending a full tablet report...
I20260809 05:22:01.575331 2276 ts_tablet_manager.cc:1428] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260809 05:22:01.576050 2278 raft_consensus.cc:348] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.576050 2277 raft_consensus.cc:348] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.576504 2277 raft_consensus.cc:374] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.576519 2278 raft_consensus.cc:374] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.576609 2277 raft_consensus.cc:729] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.576609 2278 raft_consensus.cc:729] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.577291 2278 consensus_queue.cc:260] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.577340 2277 consensus_queue.cc:260] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.579836 2278 ts_tablet_manager.cc:1428] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 05:22:01.579945 2277 ts_tablet_manager.cc:1428] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260809 05:22:01.580363 2236 heartbeater.cc:503] Master 127.22.1.126:46805 was elected leader, sending a full tablet report...
W20260809 05:22:01.599543 1824 tablet.cc:2369] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:22:01.627152 2284 raft_consensus.cc:482] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:01.627532 2284 raft_consensus.cc:504] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.629195 2284 leader_election.cc:288] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427), 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757)
W20260809 05:22:01.632455 2024 tablet.cc:2369] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:22:01.636502 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.629588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:43113 (local address 127.22.1.67:40427)
0809 05:22:01.629800 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.629806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.630430 (+ 624us) server_negotiation.cc:407] Connection header received
0809 05:22:01.630629 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.630634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.630716 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.630815 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.631611 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.632685 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.634781 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.635547 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.635650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.635763 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.636134 (+ 371us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.636260 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:22:01.636509 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.629603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:46415 (local address 127.22.1.65:37757)
0809 05:22:01.629824 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.629832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.630465 (+ 633us) server_negotiation.cc:407] Connection header received
0809 05:22:01.630683 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.630687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.630775 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.630860 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.631594 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.632709 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.634786 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.635566 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.635700 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.635800 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.636137 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.636265 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:22:01.636670 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.629578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.22.1.66:46415)
0809 05:22:01.630371 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.630385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.630560 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.630876 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.630880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.630897 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.631378 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.631393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.632872 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.632876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.634627 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.634641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.635804 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.635810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.635968 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.636011 (+ 43us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.636092 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.636218 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":160,"spinlock_wait_cycles":27520,"thread_start_us":501,"threads_started":1}
I20260809 05:22:01.636999 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.629455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40427 (local address 127.22.1.66:43113)
0809 05:22:01.630293 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.630318 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.630482 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.630849 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.630854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.630882 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.631484 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.631495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.632822 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.632827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.634621 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.634634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.635775 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.635785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.635958 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.636002 (+ 44us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.636080 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.636208 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":57,"spinlock_wait_cycles":26368,"thread_start_us":382,"threads_started":1}
I20260809 05:22:01.637666 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c86bd130464501b09bd776ed8cbcf7" candidate_uuid: "09a0db971ea4466eabbc49796a93cfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062" is_pre_election: true
I20260809 05:22:01.637943 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c86bd130464501b09bd776ed8cbcf7" candidate_uuid: "09a0db971ea4466eabbc49796a93cfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39" is_pre_election: true
I20260809 05:22:01.638159 1748 raft_consensus.cc:2454] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0db971ea4466eabbc49796a93cfe3 in term 0.
I20260809 05:22:01.638437 2157 raft_consensus.cc:2454] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09a0db971ea4466eabbc49796a93cfe3 in term 0.
I20260809 05:22:01.638782 1854 leader_election.cc:302] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 33f4f98f54ce453eb523f6384cb40062; no voters:
I20260809 05:22:01.639212 2284 raft_consensus.cc:2781] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:01.639372 2284 raft_consensus.cc:482] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:01.639482 2284 raft_consensus.cc:3037] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.641930 2284 raft_consensus.cc:504] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.642522 2284 leader_election.cc:288] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [CANDIDATE]: Term 1 election: Requested vote from peers 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427), 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757)
I20260809 05:22:01.643031 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c86bd130464501b09bd776ed8cbcf7" candidate_uuid: "09a0db971ea4466eabbc49796a93cfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062"
I20260809 05:22:01.643174 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c86bd130464501b09bd776ed8cbcf7" candidate_uuid: "09a0db971ea4466eabbc49796a93cfe3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39"
I20260809 05:22:01.643229 1748 raft_consensus.cc:3037] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.643368 2157 raft_consensus.cc:3037] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.645478 1748 raft_consensus.cc:2454] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0db971ea4466eabbc49796a93cfe3 in term 1.
I20260809 05:22:01.645607 2157 raft_consensus.cc:2454] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09a0db971ea4466eabbc49796a93cfe3 in term 1.
I20260809 05:22:01.645959 1854 leader_election.cc:302] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 33f4f98f54ce453eb523f6384cb40062; no voters:
I20260809 05:22:01.646245 2284 raft_consensus.cc:2781] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:01.646651 2284 raft_consensus.cc:686] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 LEADER]: Becoming Leader. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Running, Role: LEADER
I20260809 05:22:01.647099 2284 consensus_queue.cc:237] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } }
I20260809 05:22:01.651600 1485 catalog_manager.cc:5495] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "09a0db971ea4466eabbc49796a93cfe3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } health_report { overall_health: UNKNOWN } } }
W20260809 05:22:01.660252 2237 tablet.cc:2369] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:22:01.693526 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.688158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.22.1.66:44113)
0809 05:22:01.688376 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.688382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.688404 (+ 22us) server_negotiation.cc:407] Connection header received
0809 05:22:01.688673 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.688677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.688761 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.688898 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:01.689308 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.690048 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.691384 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.691803 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.691912 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.692367 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:01.692918 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:01.693052 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.693174 (+ 122us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.693258 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:22:01.701524 2157 raft_consensus.cc:1261] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Refusing update from remote peer 09a0db971ea4466eabbc49796a93cfe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:01.701527 1748 raft_consensus.cc:1261] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Refusing update from remote peer 09a0db971ea4466eabbc49796a93cfe3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:01.702418 2292 consensus_queue.cc:1035] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:01.702780 2284 consensus_queue.cc:1035] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:01.735529 1485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:01.735834 1485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:01.740265 1485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Sending DeleteTablet for 3 replicas of tablet f6c86bd130464501b09bd776ed8cbcf7
I20260809 05:22:01.741554 2129 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:01 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37244
I20260809 05:22:01.741688 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:01 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37586
I20260809 05:22:01.743148 1705 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c86bd130464501b09bd776ed8cbcf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:01 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55860
I20260809 05:22:01.744745 1485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:22:01.746268 1485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:01.756729 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d (DEFAULT_TABLE table=test-table [id=705616d059804a6187f18c8578fd1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.757015 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d (DEFAULT_TABLE table=test-table [id=705616d059804a6187f18c8578fd1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.757465 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f20a3dc09f345758db7bf2b45fcaa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.757860 2320 tablet_replica.cc:330] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: stopping tablet replica
I20260809 05:22:01.757881 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f20a3dc09f345758db7bf2b45fcaa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.758406 2320 raft_consensus.cc:2229] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:22:01.759112 2320 raft_consensus.cc:2258] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:01.760075 2321 tablet_replica.cc:330] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: stopping tablet replica
I20260809 05:22:01.760445 1707 tablet_service.cc:1467] Processing CreateTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d (DEFAULT_TABLE table=test-table [id=705616d059804a6187f18c8578fd1f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:01.760535 2321 raft_consensus.cc:2229] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:01.761016 1707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f20a3dc09f345758db7bf2b45fcaa2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:01.761060 2321 raft_consensus.cc:2258] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:01.761683 2318 tablet_replica.cc:330] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: stopping tablet replica
I20260809 05:22:01.762059 2318 raft_consensus.cc:2229] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:01.762501 2318 raft_consensus.cc:2258] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:01.764541 2276 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Bootstrap starting.
I20260809 05:22:01.764843 2318 ts_tablet_manager.cc:1905] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:01.766279 2276 tablet_bootstrap.cc:654] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.772706 2318 ts_tablet_manager.cc:1918] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:01.772873 2318 log.cc:1196] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/f6c86bd130464501b09bd776ed8cbcf7
I20260809 05:22:01.773463 2318 ts_tablet_manager.cc:1939] T f6c86bd130464501b09bd776ed8cbcf7 P 3943e990350a40ac833ccbc92b3eea39: Deleting consensus metadata
I20260809 05:22:01.775573 1469 catalog_manager.cc:4841] TS 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427): tablet f6c86bd130464501b09bd776ed8cbcf7 (table test-table [id=315f53d7e526402da08b5ff157a1e78f]) successfully deleted
I20260809 05:22:01.776830 2277 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Bootstrap starting.
I20260809 05:22:01.776895 2276 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: No bootstrap required, opened a new log
I20260809 05:22:01.777031 2276 ts_tablet_manager.cc:1397] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260809 05:22:01.778048 2276 raft_consensus.cc:348] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.778296 2276 raft_consensus.cc:374] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.778402 2276 raft_consensus.cc:729] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.778513 2277 tablet_bootstrap.cc:654] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.778782 2278 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Bootstrap starting.
I20260809 05:22:01.778774 2321 ts_tablet_manager.cc:1905] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:01.778741 2276 consensus_queue.cc:260] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.780117 2276 ts_tablet_manager.cc:1428] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260809 05:22:01.780416 2278 tablet_bootstrap.cc:654] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:01.786425 2321 ts_tablet_manager.cc:1918] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:01.786620 2321 log.cc:1196] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/f6c86bd130464501b09bd776ed8cbcf7
I20260809 05:22:01.787199 2321 ts_tablet_manager.cc:1939] T f6c86bd130464501b09bd776ed8cbcf7 P 33f4f98f54ce453eb523f6384cb40062: Deleting consensus metadata
I20260809 05:22:01.788045 2278 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: No bootstrap required, opened a new log
I20260809 05:22:01.788223 2278 ts_tablet_manager.cc:1397] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260809 05:22:01.789093 1462 catalog_manager.cc:4841] TS 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757): tablet f6c86bd130464501b09bd776ed8cbcf7 (table test-table [id=315f53d7e526402da08b5ff157a1e78f]) successfully deleted
I20260809 05:22:01.789321 2278 raft_consensus.cc:348] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.789726 2278 raft_consensus.cc:374] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.789855 2278 raft_consensus.cc:729] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.791018 2320 ts_tablet_manager.cc:1905] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:01.791967 2278 consensus_queue.cc:260] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.795435 2278 ts_tablet_manager.cc:1428] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260809 05:22:01.798950 2320 ts_tablet_manager.cc:1918] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:01.799912 2320 log.cc:1196] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/f6c86bd130464501b09bd776ed8cbcf7
I20260809 05:22:01.800417 2277 tablet_bootstrap.cc:492] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: No bootstrap required, opened a new log
I20260809 05:22:01.800501 2320 ts_tablet_manager.cc:1939] T f6c86bd130464501b09bd776ed8cbcf7 P 09a0db971ea4466eabbc49796a93cfe3: Deleting consensus metadata
I20260809 05:22:01.800585 2277 ts_tablet_manager.cc:1397] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.019s
I20260809 05:22:01.801302 2277 raft_consensus.cc:348] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.801492 2277 raft_consensus.cc:374] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:01.801599 2277 raft_consensus.cc:729] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Initialized, Role: FOLLOWER
I20260809 05:22:01.801873 2277 consensus_queue.cc:260] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.802419 1468 catalog_manager.cc:4841] TS 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113): tablet f6c86bd130464501b09bd776ed8cbcf7 (table test-table [id=315f53d7e526402da08b5ff157a1e78f]) successfully deleted
I20260809 05:22:01.802532 2277 ts_tablet_manager.cc:1428] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:22:01.870232 2286 raft_consensus.cc:482] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:01.870467 2286 raft_consensus.cc:504] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.872505 2286 leader_election.cc:288] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113), 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427)
I20260809 05:22:01.893359 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.873219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:52321 (local address 127.22.1.66:44113)
0809 05:22:01.873421 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.873427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.885013 (+ 11586us) server_negotiation.cc:407] Connection header received
0809 05:22:01.885461 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.885468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.885565 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.885633 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.886438 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.887669 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.890691 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.891978 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.892100 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.892210 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.892860 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.893007 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260809 05:22:01.894169 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.872817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:44113 (local address 127.22.1.65:52321)
0809 05:22:01.884900 (+ 12083us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.884921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.885087 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.885850 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.885855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.885888 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.886237 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.886248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.888181 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.888186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.890507 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.890521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.892414 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.892420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.892660 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.892696 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.893636 (+ 940us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.893770 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11932,"mutex_wait_us":3,"thread_start_us":16189,"threads_started":1}
I20260809 05:22:01.895016 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f20a3dc09f345758db7bf2b45fcaa2d" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3" is_pre_election: true
I20260809 05:22:01.895524 1958 raft_consensus.cc:2454] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 0.
I20260809 05:22:01.896061 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.873640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:42075 (local address 127.22.1.67:40427)
0809 05:22:01.873816 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.873821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.889482 (+ 15661us) server_negotiation.cc:407] Connection header received
0809 05:22:01.889697 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.889703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.889778 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.889835 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:01.890691 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.891732 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.894103 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.895103 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.895217 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.895310 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.895670 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.895802 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:22:01.896278 1629 leader_election.cc:302] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 33f4f98f54ce453eb523f6384cb40062; no voters:
I20260809 05:22:01.896770 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.873003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40427 (local address 127.22.1.65:42075)
0809 05:22:01.889367 (+ 16364us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.889392 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.889513 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.890086 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.890091 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.890117 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:01.890520 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.890531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.891913 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.891919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.893902 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.893916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.895310 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.895317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.895481 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.895526 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.896244 (+ 718us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.896390 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16201,"mutex_wait_us":30,"thread_start_us":12152,"threads_started":1}
I20260809 05:22:01.896847 2286 raft_consensus.cc:2781] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:01.896973 2286 raft_consensus.cc:482] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:01.897063 2286 raft_consensus.cc:3037] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.899487 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f20a3dc09f345758db7bf2b45fcaa2d" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39" is_pre_election: true
I20260809 05:22:01.899742 2157 raft_consensus.cc:2454] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 0.
I20260809 05:22:01.900180 2286 raft_consensus.cc:504] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.900864 2286 leader_election.cc:288] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 election: Requested vote from peers 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113), 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427)
I20260809 05:22:01.901401 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f20a3dc09f345758db7bf2b45fcaa2d" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3"
I20260809 05:22:01.901556 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f20a3dc09f345758db7bf2b45fcaa2d" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39"
I20260809 05:22:01.901633 1958 raft_consensus.cc:3037] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.901758 2157 raft_consensus.cc:3037] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:01.904618 2157 raft_consensus.cc:2454] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 1.
I20260809 05:22:01.905064 1632 leader_election.cc:302] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f4f98f54ce453eb523f6384cb40062, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:01.905556 2286 raft_consensus.cc:2781] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:01.906646 1958 raft_consensus.cc:2454] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 1.
I20260809 05:22:01.907325 2286 raft_consensus.cc:686] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 1 LEADER]: Becoming Leader. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Running, Role: LEADER
I20260809 05:22:01.907714 2286 consensus_queue.cc:237] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:01.911260 1485 catalog_manager.cc:5495] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } health_report { overall_health: UNKNOWN } } }
Aug 09 05:22:01 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252919, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.126@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:01.986786 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.967736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.22.1.126:46805)
0809 05:22:01.967953 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:01.967959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:01.968510 (+ 551us) server_negotiation.cc:407] Connection header received
0809 05:22:01.968664 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:01.968670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:01.968754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:01.968888 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:01.970065 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.971293 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.972623 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.973296 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.973428 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.981041 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:01.981048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:01.981053 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:01.981087 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:01.983678 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.984639 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.984643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.984645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.984770 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:01.985234 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:01.985237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:01.985239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:01.985522 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:01.985651 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:01.986152 (+ 501us) server_negotiation.cc:299] Negotiation successful
0809 05:22:01.986484 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":152,"server-negotiator.queue_time_us":73}
I20260809 05:22:01.986831 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:01.967736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:46805 (local address 127.0.0.1:38978)
0809 05:22:01.968396 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:01.968414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:01.968545 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:01.968896 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:01.968899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:01.969637 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:01.969905 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.969915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.971457 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.971461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.972445 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:01.972462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:01.973479 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:01.973484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:01.973621 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:01.978039 (+ 4418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:01.978071 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:01.980809 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:01.983830 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.983843 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.983851 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.984500 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.984879 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:01.984884 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:01.984888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:01.985070 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:01.985655 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:01.985664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:01.985875 (+ 211us) client_negotiation.cc:769] Sending connection context
0809 05:22:01.986048 (+ 173us) client_negotiation.cc:240] Negotiation successful
0809 05:22:01.986371 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":344,"threads_started":1}
I20260809 05:22:01.992522 1485 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:38978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260809 05:22:01.993871 1485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:02.001062 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9ea89e329d4774b650de18d16a44b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d6483056b54055b672af7b74c4cf9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:22:02.001801 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9ea89e329d4774b650de18d16a44b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.001899 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9ea89e329d4774b650de18d16a44b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d6483056b54055b672af7b74c4cf9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:22:02.002401 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9ea89e329d4774b650de18d16a44b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.001062 1707 tablet_service.cc:1467] Processing CreateTablet for tablet 2b9ea89e329d4774b650de18d16a44b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=87d6483056b54055b672af7b74c4cf9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260809 05:22:02.005878 1707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b9ea89e329d4774b650de18d16a44b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.009451 2278 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: Bootstrap starting.
I20260809 05:22:02.009454 2276 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: Bootstrap starting.
I20260809 05:22:02.011179 2278 tablet_bootstrap.cc:654] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.011345 2276 tablet_bootstrap.cc:654] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.013211 2277 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: Bootstrap starting.
I20260809 05:22:02.014889 2277 tablet_bootstrap.cc:654] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.017194 2277 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: No bootstrap required, opened a new log
I20260809 05:22:02.017443 2277 ts_tablet_manager.cc:1397] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260809 05:22:02.019474 2277 raft_consensus.cc:348] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.019814 2277 raft_consensus.cc:374] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.019940 2277 raft_consensus.cc:729] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.020251 2277 consensus_queue.cc:260] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.020944 2277 ts_tablet_manager.cc:1428] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260809 05:22:02.020804 2286 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.021752 2286 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.022986 2278 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: No bootstrap required, opened a new log
I20260809 05:22:02.023178 2278 ts_tablet_manager.cc:1397] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260809 05:22:02.023929 2278 raft_consensus.cc:348] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.024187 2278 raft_consensus.cc:374] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.024266 2278 raft_consensus.cc:729] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.024634 2276 tablet_bootstrap.cc:492] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: No bootstrap required, opened a new log
I20260809 05:22:02.024544 2278 consensus_queue.cc:260] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.024811 2276 ts_tablet_manager.cc:1397] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260809 05:22:02.024982 2285 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.025307 2285 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.025660 2276 raft_consensus.cc:348] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.025928 2276 raft_consensus.cc:374] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.026006 2276 raft_consensus.cc:729] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.026237 2278 ts_tablet_manager.cc:1428] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260809 05:22:02.026407 2276 consensus_queue.cc:260] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.027037 2276 ts_tablet_manager.cc:1428] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:22:02.026934 2310 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.027374 2310 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.114727 2286 raft_consensus.cc:482] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:02.114929 2286 raft_consensus.cc:504] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.115634 2286 leader_election.cc:288] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113), 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427)
I20260809 05:22:02.116120 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9ea89e329d4774b650de18d16a44b3" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3" is_pre_election: true
I20260809 05:22:02.116320 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9ea89e329d4774b650de18d16a44b3" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39" is_pre_election: true
I20260809 05:22:02.116425 1958 raft_consensus.cc:2454] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 0.
I20260809 05:22:02.116573 2157 raft_consensus.cc:2454] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 0.
I20260809 05:22:02.116905 1629 leader_election.cc:302] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 33f4f98f54ce453eb523f6384cb40062; no voters:
I20260809 05:22:02.117223 2286 raft_consensus.cc:2781] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:02.117326 2286 raft_consensus.cc:482] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:02.117443 2286 raft_consensus.cc:3037] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.119416 2286 raft_consensus.cc:504] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.120004 2286 leader_election.cc:288] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 election: Requested vote from peers 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113), 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427)
I20260809 05:22:02.120481 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9ea89e329d4774b650de18d16a44b3" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3"
I20260809 05:22:02.120600 2157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b9ea89e329d4774b650de18d16a44b3" candidate_uuid: "33f4f98f54ce453eb523f6384cb40062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3943e990350a40ac833ccbc92b3eea39"
I20260809 05:22:02.120754 1958 raft_consensus.cc:3037] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.120821 2157 raft_consensus.cc:3037] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.123687 2157 raft_consensus.cc:2454] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 1.
I20260809 05:22:02.123870 1958 raft_consensus.cc:2454] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33f4f98f54ce453eb523f6384cb40062 in term 1.
I20260809 05:22:02.124152 1632 leader_election.cc:302] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f4f98f54ce453eb523f6384cb40062, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:02.124475 2286 raft_consensus.cc:2781] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:02.124663 2286 raft_consensus.cc:686] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [term 1 LEADER]: Becoming Leader. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Running, Role: LEADER
I20260809 05:22:02.124924 2286 consensus_queue.cc:237] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.126696 2328 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: TxnStatusTablet state changed. Reason: New leader 33f4f98f54ce453eb523f6384cb40062. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.127116 2328 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:22:02.128330 1485 catalog_manager.cc:5495] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } health_report { overall_health: UNKNOWN } } }
I20260809 05:22:02.133019 2350 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260809 05:22:02.133268 2350 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260809 05:22:02.147382 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.141219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.0.0.1:55872)
0809 05:22:02.141583 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.141607 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.141756 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.142719 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.142725 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.142737 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:22:02.142906 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.142914 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.143976 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.143982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.144993 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.145006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.145768 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.145774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.145946 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.146002 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.146856 (+ 854us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.146876 (+ 20us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.146956 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.147103 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260809 05:22:02.147410 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.141305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.22.1.65:37757)
0809 05:22:02.142366 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.142370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.142385 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:22:02.142453 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.142457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.142542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.142671 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.143080 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.143851 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.145142 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.145626 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.145743 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.146214 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.146748 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.146874 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.146986 (+ 112us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.147075 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":851,"threads_started":1}
I20260809 05:22:02.151954 2157 raft_consensus.cc:1261] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Refusing update from remote peer 33f4f98f54ce453eb523f6384cb40062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.152204 1958 raft_consensus.cc:1261] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Refusing update from remote peer 33f4f98f54ce453eb523f6384cb40062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.152930 2328 consensus_queue.cc:1035] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.153231 2286 consensus_queue.cc:1035] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.157547 2310 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: TxnStatusTablet state changed. Reason: New leader 33f4f98f54ce453eb523f6384cb40062. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.157821 2310 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.162693 2286 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.163030 2286 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:22:02.164845 2298 mvcc.cc:205] Tried to move back new op lower bound from 7316491969126944768 to 7316491969026469888. Current Snapshot: MvccSnapshot[applied={T|T < 7316491969126944768}]
I20260809 05:22:02.165145 2310 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.165414 2310 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 09a0db971ea4466eabbc49796a93cfe3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.170783 2285 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: TxnStatusTablet state changed. Reason: New leader 33f4f98f54ce453eb523f6384cb40062. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.171037 2285 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.173939 2328 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.174247 2328 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 33f4f98f54ce453eb523f6384cb40062: This TxnStatusTablet replica's current role is: LEADER
I20260809 05:22:02.177402 2285 tablet_replica.cc:439] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "33f4f98f54ce453eb523f6384cb40062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } } }
I20260809 05:22:02.177662 2285 tablet_replica.cc:442] T 2b9ea89e329d4774b650de18d16a44b3 P 3943e990350a40ac833ccbc92b3eea39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260809 05:22:02.183362 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.171355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.22.1.65:37757)
0809 05:22:02.171515 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.171523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.171543 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:22:02.171608 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.171612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.171689 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.171768 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.177130 (+ 5362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.178496 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.180503 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.181205 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.181933 (+ 728us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.182057 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.182190 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.182264 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.182897 (+ 633us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.182990 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 05:22:02.208504 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.202798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56619 (local address 127.22.1.65:37757)
0809 05:22:02.202991 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.202996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.203617 (+ 621us) server_negotiation.cc:407] Connection header received
0809 05:22:02.203814 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.203819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.203878 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.203962 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.204486 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.205442 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.206869 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.207297 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.207398 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.207825 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.207918 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.207998 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.208115 (+ 117us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.208179 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:22:02.208809 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.202805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.22.1.65:56619)
0809 05:22:02.203523 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.203539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.203674 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.203974 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.203978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.203988 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:22:02.204297 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.204306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.205581 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.205587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.206744 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.206757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.207474 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.207479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.207613 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.207663 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.208008 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.208025 (+ 17us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.208091 (+ 66us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.208214 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":419,"threads_started":1}
I20260809 05:22:02.219628 2157 raft_consensus.cc:1261] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Refusing update from remote peer 33f4f98f54ce453eb523f6384cb40062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.219628 1958 raft_consensus.cc:1261] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Refusing update from remote peer 33f4f98f54ce453eb523f6384cb40062: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.220229 2328 consensus_queue.cc:1035] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.220683 2328 consensus_queue.cc:1035] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.280593 1485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.280833 1485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.283783 1485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Sending DeleteTablet for 3 replicas of tablet 4f20a3dc09f345758db7bf2b45fcaa2d
I20260809 05:22:02.284559 1705 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55860
I20260809 05:22:02.284795 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37586
I20260809 05:22:02.284893 2128 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f20a3dc09f345758db7bf2b45fcaa2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37244
I20260809 05:22:02.285329 2320 tablet_replica.cc:330] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: stopping tablet replica
I20260809 05:22:02.285467 2368 tablet_replica.cc:330] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: stopping tablet replica
I20260809 05:22:02.285553 2320 raft_consensus.cc:2229] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.285647 2368 raft_consensus.cc:2229] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.285769 2320 raft_consensus.cc:2258] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.285869 2368 raft_consensus.cc:2258] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.286054 2321 tablet_replica.cc:330] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: stopping tablet replica
I20260809 05:22:02.286269 2321 raft_consensus.cc:2229] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:22:02.286767 2321 raft_consensus.cc:2258] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.286789 2320 ts_tablet_manager.cc:1905] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.287928 2321 ts_tablet_manager.cc:1905] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.293534 2320 ts_tablet_manager.cc:1918] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.293702 2320 log.cc:1196] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/4f20a3dc09f345758db7bf2b45fcaa2d
I20260809 05:22:02.294114 2320 ts_tablet_manager.cc:1939] T 4f20a3dc09f345758db7bf2b45fcaa2d P 09a0db971ea4466eabbc49796a93cfe3: Deleting consensus metadata
I20260809 05:22:02.294591 2368 ts_tablet_manager.cc:1905] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.295212 1468 catalog_manager.cc:4841] TS 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113): tablet 4f20a3dc09f345758db7bf2b45fcaa2d (table test-table [id=705616d059804a6187f18c8578fd1f13]) successfully deleted
I20260809 05:22:02.301504 2368 ts_tablet_manager.cc:1918] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.301664 2368 log.cc:1196] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/4f20a3dc09f345758db7bf2b45fcaa2d
I20260809 05:22:02.302093 2368 ts_tablet_manager.cc:1939] T 4f20a3dc09f345758db7bf2b45fcaa2d P 3943e990350a40ac833ccbc92b3eea39: Deleting consensus metadata
I20260809 05:22:02.303381 1469 catalog_manager.cc:4841] TS 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427): tablet 4f20a3dc09f345758db7bf2b45fcaa2d (table test-table [id=705616d059804a6187f18c8578fd1f13]) successfully deleted
I20260809 05:22:02.304831 2321 ts_tablet_manager.cc:1918] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.304999 2321 log.cc:1196] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/4f20a3dc09f345758db7bf2b45fcaa2d
I20260809 05:22:02.305447 2321 ts_tablet_manager.cc:1939] T 4f20a3dc09f345758db7bf2b45fcaa2d P 33f4f98f54ce453eb523f6384cb40062: Deleting consensus metadata
I20260809 05:22:02.306622 1462 catalog_manager.cc:4841] TS 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757): tablet 4f20a3dc09f345758db7bf2b45fcaa2d (table test-table [id=705616d059804a6187f18c8578fd1f13]) successfully deleted
Aug 09 05:22:02 dist-test-slave-rmhg krb5kdc[1412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 09 05:22:02 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:02 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 05:22:02.358021 1637 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:02.347719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.22.1.126:46805)
0809 05:22:02.347926 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.347933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.348482 (+ 549us) server_negotiation.cc:407] Connection header received
0809 05:22:02.348674 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.348680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.348754 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.348842 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:02.350219 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.350949 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.352397 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.352864 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.352977 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.357837 (+ 4860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38986: BlockingRecv error: recv got EOF from 127.0.0.1:38986 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 05:22:02 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM
I20260809 05:22:02.375838 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.361446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39000 (local address 127.22.1.126:46805)
0809 05:22:02.361610 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.361614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.362235 (+ 621us) server_negotiation.cc:407] Connection header received
0809 05:22:02.362390 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.362394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.362456 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.362545 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:02.363663 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.364340 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.365620 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.366111 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.366221 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.369891 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:02.369896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:02.369899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:02.369932 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:02.373077 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:02.373863 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:02.373869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:02.373873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:02.374000 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:02.374349 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:02.374354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:02.374358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:02.374634 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:02.374854 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.375190 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.375426 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:22:02.391901 1485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:22:02.392640 1485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:02.400084 1705 tablet_service.cc:1467] Processing CreateTablet for tablet 204ecd60f2044142be4856c4883cc43a (DEFAULT_TABLE table=test-table [id=388ca1cec9594a3f96d059e9fa637fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.400343 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 204ecd60f2044142be4856c4883cc43a (DEFAULT_TABLE table=test-table [id=388ca1cec9594a3f96d059e9fa637fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.400614 1705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ecd60f2044142be4856c4883cc43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.400774 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ecd60f2044142be4856c4883cc43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.404186 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 204ecd60f2044142be4856c4883cc43a (DEFAULT_TABLE table=test-table [id=388ca1cec9594a3f96d059e9fa637fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.404711 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 204ecd60f2044142be4856c4883cc43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.407208 2277 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Bootstrap starting.
I20260809 05:22:02.408896 2277 tablet_bootstrap.cc:654] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.411981 2278 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Bootstrap starting.
I20260809 05:22:02.413681 2278 tablet_bootstrap.cc:654] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.414237 2277 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: No bootstrap required, opened a new log
I20260809 05:22:02.414454 2277 ts_tablet_manager.cc:1397] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260809 05:22:02.415282 2277 raft_consensus.cc:348] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.415678 2277 raft_consensus.cc:374] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.415822 2277 raft_consensus.cc:729] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.416122 2277 consensus_queue.cc:260] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.416791 2277 ts_tablet_manager.cc:1428] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260809 05:22:02.417603 2276 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Bootstrap starting.
I20260809 05:22:02.419215 2276 tablet_bootstrap.cc:654] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.426810 2278 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: No bootstrap required, opened a new log
I20260809 05:22:02.427057 2278 ts_tablet_manager.cc:1397] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260809 05:22:02.427989 2278 raft_consensus.cc:348] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.428278 2278 raft_consensus.cc:374] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.428380 2278 raft_consensus.cc:729] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.428620 2278 consensus_queue.cc:260] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.429234 2278 ts_tablet_manager.cc:1428] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:22:02.433501 2276 tablet_bootstrap.cc:492] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: No bootstrap required, opened a new log
I20260809 05:22:02.433733 2276 ts_tablet_manager.cc:1397] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20260809 05:22:02.434404 2276 raft_consensus.cc:348] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.434850 2276 raft_consensus.cc:374] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.434952 2276 raft_consensus.cc:729] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.435214 2276 consensus_queue.cc:260] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.435866 2276 ts_tablet_manager.cc:1428] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:22:02.488185 2285 raft_consensus.cc:482] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:02.488482 2285 raft_consensus.cc:504] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.490367 2285 leader_election.cc:288] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757), 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113)
I20260809 05:22:02.508826 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.490973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:39815 (local address 127.22.1.66:44113)
0809 05:22:02.501363 (+ 10390us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.501370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.502483 (+ 1113us) server_negotiation.cc:407] Connection header received
0809 05:22:02.502602 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.502607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.502665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.502755 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:02.503563 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.504633 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.507123 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.507869 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.507983 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.508090 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.508385 (+ 295us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.508532 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10215,"thread_start_us":10395,"threads_started":1}
I20260809 05:22:02.508844 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.490741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:44113 (local address 127.22.1.67:39815)
0809 05:22:02.502373 (+ 11632us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.502390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.502496 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.502937 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.502942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.502975 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:02.503367 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.503383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.504807 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.504811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.506960 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.506973 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.508061 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.508068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.508240 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.508263 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.508339 (+ 76us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.508460 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11520,"mutex_wait_us":26,"thread_start_us":10948,"threads_started":1}
I20260809 05:22:02.509828 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ecd60f2044142be4856c4883cc43a" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3" is_pre_election: true
I20260809 05:22:02.509995 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.491293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:52737 (local address 127.22.1.65:37757)
0809 05:22:02.491530 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.491536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.502132 (+ 10596us) server_negotiation.cc:407] Connection header received
0809 05:22:02.502483 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.502489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.502595 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.502717 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:02.503231 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.504410 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.507245 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.509091 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.509268 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.509378 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.509684 (+ 306us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.509794 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:22:02.510111 1958 raft_consensus.cc:2454] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 0.
I20260809 05:22:02.510733 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.490671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.22.1.67:52737)
0809 05:22:02.501994 (+ 11323us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.502017 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.502280 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.502739 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.502743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.502770 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:02.503058 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.503068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.504570 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.504585 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.507037 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.507055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.509330 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.509337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.509493 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.509525 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.510231 (+ 706us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.510362 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11180,"mutex_wait_us":18,"thread_start_us":11403,"threads_started":1}
I20260809 05:22:02.510707 2056 leader_election.cc:302] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:02.511161 2285 raft_consensus.cc:2781] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:02.511268 2285 raft_consensus.cc:482] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:02.511358 2285 raft_consensus.cc:3037] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.511691 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ecd60f2044142be4856c4883cc43a" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062" is_pre_election: true
I20260809 05:22:02.511942 1748 raft_consensus.cc:2454] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 0.
I20260809 05:22:02.513643 2285 raft_consensus.cc:504] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.514158 2285 leader_election.cc:288] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 election: Requested vote from peers 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757), 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113)
I20260809 05:22:02.514632 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ecd60f2044142be4856c4883cc43a" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3"
I20260809 05:22:02.514642 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "204ecd60f2044142be4856c4883cc43a" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062"
I20260809 05:22:02.514809 1958 raft_consensus.cc:3037] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.514853 1748 raft_consensus.cc:3037] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.516711 1958 raft_consensus.cc:2454] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 1.
I20260809 05:22:02.517079 2056 leader_election.cc:302] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09a0db971ea4466eabbc49796a93cfe3, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:02.517361 2285 raft_consensus.cc:2781] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:02.517793 2285 raft_consensus.cc:686] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 1 LEADER]: Becoming Leader. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Running, Role: LEADER
I20260809 05:22:02.517920 1748 raft_consensus.cc:2454] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 1.
I20260809 05:22:02.518218 2285 consensus_queue.cc:237] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.521507 1485 catalog_manager.cc:5495] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "3943e990350a40ac833ccbc92b3eea39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } health_report { overall_health: HEALTHY } } }
I20260809 05:22:02.537489 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.530749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.22.1.67:40427)
0809 05:22:02.531198 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.531204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.531225 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:22:02.531347 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.531352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.531465 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.531669 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.532038 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.533278 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.534626 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.535239 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.535380 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.535836 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.536704 (+ 868us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.536869 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.536977 (+ 108us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.537069 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":209,"threads_started":1}
I20260809 05:22:02.541636 1748 raft_consensus.cc:1261] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Refusing update from remote peer 3943e990350a40ac833ccbc92b3eea39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.541635 1958 raft_consensus.cc:1261] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Refusing update from remote peer 3943e990350a40ac833ccbc92b3eea39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.542447 2391 consensus_queue.cc:1035] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.542788 2285 consensus_queue.cc:1035] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.548979 2304 mvcc.cc:205] Tried to move back new op lower bound from 7316491970722693120 to 7316491970638315520. Current Snapshot: MvccSnapshot[applied={T|T < 7316491970722693120}]
I20260809 05:22:02.551062 2303 mvcc.cc:205] Tried to move back new op lower bound from 7316491970722693120 to 7316491970638315520. Current Snapshot: MvccSnapshot[applied={T|T < 7316491970722693120}]
I20260809 05:22:02.565029 1485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.565340 1485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.569350 1485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Sending DeleteTablet for 3 replicas of tablet 204ecd60f2044142be4856c4883cc43a
I20260809 05:22:02.570443 1705 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ecd60f2044142be4856c4883cc43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55860
I20260809 05:22:02.570845 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ecd60f2044142be4856c4883cc43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37586
I20260809 05:22:02.571194 2321 tablet_replica.cc:330] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: stopping tablet replica
I20260809 05:22:02.571415 2321 raft_consensus.cc:2229] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.571580 2320 tablet_replica.cc:330] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: stopping tablet replica
I20260809 05:22:02.571712 2321 raft_consensus.cc:2258] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.571781 2320 raft_consensus.cc:2229] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.572063 2320 raft_consensus.cc:2258] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.572347 1485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 05:22:02.572922 2321 ts_tablet_manager.cc:1905] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260809 05:22:02.573020 1485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:02.573644 2128 tablet_service.cc:1514] Processing DeleteTablet for tablet 204ecd60f2044142be4856c4883cc43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37244
I20260809 05:22:02.573951 2368 tablet_replica.cc:330] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: stopping tablet replica
I20260809 05:22:02.574154 2368 raft_consensus.cc:2229] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:22:02.574712 2368 raft_consensus.cc:2258] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.576282 2320 ts_tablet_manager.cc:1905] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.577132 2368 ts_tablet_manager.cc:1905] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.584508 2368 ts_tablet_manager.cc:1918] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:02.584687 2368 log.cc:1196] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/204ecd60f2044142be4856c4883cc43a
I20260809 05:22:02.585119 2368 ts_tablet_manager.cc:1939] T 204ecd60f2044142be4856c4883cc43a P 3943e990350a40ac833ccbc92b3eea39: Deleting consensus metadata
I20260809 05:22:02.586311 2321 ts_tablet_manager.cc:1918] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:02.586473 2321 log.cc:1196] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/204ecd60f2044142be4856c4883cc43a
I20260809 05:22:02.586423 1469 catalog_manager.cc:4841] TS 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427): tablet 204ecd60f2044142be4856c4883cc43a (table test-table [id=388ca1cec9594a3f96d059e9fa637fc9]) successfully deleted
I20260809 05:22:02.586925 2321 ts_tablet_manager.cc:1939] T 204ecd60f2044142be4856c4883cc43a P 33f4f98f54ce453eb523f6384cb40062: Deleting consensus metadata
I20260809 05:22:02.588097 1462 catalog_manager.cc:4841] TS 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757): tablet 204ecd60f2044142be4856c4883cc43a (table test-table [id=388ca1cec9594a3f96d059e9fa637fc9]) successfully deleted
I20260809 05:22:02.591398 1705 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7267d32e8454193a87c39dc095664 (DEFAULT_TABLE table=test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.591653 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7267d32e8454193a87c39dc095664 (DEFAULT_TABLE table=test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.591856 1705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a7267d32e8454193a87c39dc095664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.591825 2128 tablet_service.cc:1467] Processing CreateTablet for tablet 27a7267d32e8454193a87c39dc095664 (DEFAULT_TABLE table=test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:02.592115 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a7267d32e8454193a87c39dc095664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.592247 2128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27a7267d32e8454193a87c39dc095664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:02.597967 2277 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Bootstrap starting.
I20260809 05:22:02.598328 2276 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Bootstrap starting.
I20260809 05:22:02.599013 2278 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Bootstrap starting.
I20260809 05:22:02.599555 2277 tablet_bootstrap.cc:654] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.599982 2276 tablet_bootstrap.cc:654] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.600656 2278 tablet_bootstrap.cc:654] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:02.590533 2320 ts_tablet_manager.cc:1918] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:02.603948 2320 log.cc:1196] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/204ecd60f2044142be4856c4883cc43a
I20260809 05:22:02.604451 2320 ts_tablet_manager.cc:1939] T 204ecd60f2044142be4856c4883cc43a P 09a0db971ea4466eabbc49796a93cfe3: Deleting consensus metadata
I20260809 05:22:02.605616 1468 catalog_manager.cc:4841] TS 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113): tablet 204ecd60f2044142be4856c4883cc43a (table test-table [id=388ca1cec9594a3f96d059e9fa637fc9]) successfully deleted
I20260809 05:22:02.611773 2277 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: No bootstrap required, opened a new log
I20260809 05:22:02.611979 2277 ts_tablet_manager.cc:1397] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20260809 05:22:02.612831 2277 raft_consensus.cc:348] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.613107 2277 raft_consensus.cc:374] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.613221 2277 raft_consensus.cc:729] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f4f98f54ce453eb523f6384cb40062, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.613509 2277 consensus_queue.cc:260] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.614248 2278 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: No bootstrap required, opened a new log
I20260809 05:22:02.614713 2278 ts_tablet_manager.cc:1397] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260809 05:22:02.615172 2277 ts_tablet_manager.cc:1428] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:02.615517 2278 raft_consensus.cc:348] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.615716 2278 raft_consensus.cc:374] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.615794 2278 raft_consensus.cc:729] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.616694 2278 consensus_queue.cc:260] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.617277 2278 ts_tablet_manager.cc:1428] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260809 05:22:02.618729 2276 tablet_bootstrap.cc:492] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: No bootstrap required, opened a new log
I20260809 05:22:02.618925 2276 ts_tablet_manager.cc:1397] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.004s
I20260809 05:22:02.619771 2276 raft_consensus.cc:348] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.620019 2276 raft_consensus.cc:374] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:02.620091 2276 raft_consensus.cc:729] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09a0db971ea4466eabbc49796a93cfe3, State: Initialized, Role: FOLLOWER
I20260809 05:22:02.620366 2276 consensus_queue.cc:260] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.620945 2276 ts_tablet_manager.cc:1428] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260809 05:22:02.628338 2400 raft_consensus.cc:482] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:02.628609 2400 raft_consensus.cc:504] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.629539 2400 leader_election.cc:288] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757), 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113)
I20260809 05:22:02.630066 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7267d32e8454193a87c39dc095664" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062" is_pre_election: true
I20260809 05:22:02.630235 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7267d32e8454193a87c39dc095664" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3" is_pre_election: true
I20260809 05:22:02.630327 1748 raft_consensus.cc:2454] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 0.
I20260809 05:22:02.630466 1958 raft_consensus.cc:2454] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 0.
I20260809 05:22:02.630802 2054 leader_election.cc:302] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f4f98f54ce453eb523f6384cb40062, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:02.631083 2400 raft_consensus.cc:2781] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:02.631208 2400 raft_consensus.cc:482] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:02.631297 2400 raft_consensus.cc:3037] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.633531 2400 raft_consensus.cc:504] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.634184 2400 leader_election.cc:288] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 election: Requested vote from peers 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757), 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113)
I20260809 05:22:02.634521 1958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7267d32e8454193a87c39dc095664" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09a0db971ea4466eabbc49796a93cfe3"
I20260809 05:22:02.634611 1748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27a7267d32e8454193a87c39dc095664" candidate_uuid: "3943e990350a40ac833ccbc92b3eea39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f4f98f54ce453eb523f6384cb40062"
I20260809 05:22:02.634734 1958 raft_consensus.cc:3037] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.634826 1748 raft_consensus.cc:3037] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:02.637737 1958 raft_consensus.cc:2454] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 1.
I20260809 05:22:02.637782 1748 raft_consensus.cc:2454] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3943e990350a40ac833ccbc92b3eea39 in term 1.
I20260809 05:22:02.638193 2054 leader_election.cc:302] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f4f98f54ce453eb523f6384cb40062, 3943e990350a40ac833ccbc92b3eea39; no voters:
I20260809 05:22:02.638707 2400 raft_consensus.cc:2781] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:02.638872 2400 raft_consensus.cc:686] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 1 LEADER]: Becoming Leader. State: Replica: 3943e990350a40ac833ccbc92b3eea39, State: Running, Role: LEADER
I20260809 05:22:02.639178 2400 consensus_queue.cc:237] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } }
I20260809 05:22:02.642227 1485 catalog_manager.cc:5495] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "3943e990350a40ac833ccbc92b3eea39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3943e990350a40ac833ccbc92b3eea39" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40427 } health_report { overall_health: HEALTHY } } }
I20260809 05:22:02.676414 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.670486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:38405 (local address 127.22.1.65:37757)
0809 05:22:02.670685 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.670692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.671336 (+ 644us) server_negotiation.cc:407] Connection header received
0809 05:22:02.671542 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.671548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.671618 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.671692 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.672204 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.673005 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.674534 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.674982 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.675108 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.675668 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.675792 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.675865 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.676014 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.676105 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 05:22:02.676532 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.670508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37757 (local address 127.22.1.67:38405)
0809 05:22:02.671234 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.671250 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.671403 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.671725 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.671729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.671742 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:22:02.672055 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.672064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.673139 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.673145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.674384 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.674397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.675266 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.675272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.675429 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.675487 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.675893 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.675904 (+ 11us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.675984 (+ 80us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.676141 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":390,"threads_started":1}
I20260809 05:22:02.687177 1958 raft_consensus.cc:1261] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Refusing update from remote peer 3943e990350a40ac833ccbc92b3eea39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.687703 1748 raft_consensus.cc:1261] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Refusing update from remote peer 3943e990350a40ac833ccbc92b3eea39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:02.688254 2391 consensus_queue.cc:1035] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f4f98f54ce453eb523f6384cb40062" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.688506 2400 consensus_queue.cc:1035] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09a0db971ea4466eabbc49796a93cfe3" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 44113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:02.770901 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.746960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:35181 (local address 127.22.1.67:40427)
0809 05:22:02.747159 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.747166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.760603 (+ 13437us) server_negotiation.cc:407] Connection header received
0809 05:22:02.760899 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.760905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.760995 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.761060 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:02.762112 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.763326 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.768448 (+ 5122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.769135 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.769284 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.770019 (+ 735us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.770188 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.770264 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.770479 (+ 215us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.770569 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:22:02.771909 2416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.747157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40427 (local address 127.22.1.65:35181)
0809 05:22:02.760478 (+ 13321us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:02.760500 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:02.760721 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:02.761351 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:02.761356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:02.761370 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0809 05:22:02.761848 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.761871 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.763550 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.763556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.768258 (+ 4702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:02.768277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.769550 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.769558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:02.769769 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.769823 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0809 05:22:02.770330 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0809 05:22:02.770340 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 05:22:02.771359 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0809 05:22:02.771535 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13145,"thread_start_us":12926,"threads_started":1}
I20260809 05:22:02.818737 1485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.818993 1485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:02.821898 1485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11cc1d0bfe6a4ae08159d68af2330864: Sending DeleteTablet for 3 replicas of tablet 27a7267d32e8454193a87c39dc095664
I20260809 05:22:02.823112 2128 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7267d32e8454193a87c39dc095664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37244
I20260809 05:22:02.823088 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7267d32e8454193a87c39dc095664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37586
I20260809 05:22:02.823387 2320 tablet_replica.cc:330] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: stopping tablet replica
I20260809 05:22:02.823414 2368 tablet_replica.cc:330] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: stopping tablet replica
I20260809 05:22:02.823583 2368 raft_consensus.cc:2229] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:22:02.823611 2320 raft_consensus.cc:2229] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.823807 2320 raft_consensus.cc:2258] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.823930 2368 raft_consensus.cc:2258] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.825083 2320 ts_tablet_manager.cc:1905] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.825347 1705 tablet_service.cc:1514] Processing DeleteTablet for tablet 27a7267d32e8454193a87c39dc095664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:02 UTC) from {username='oryx', principal='oryx/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55860
I20260809 05:22:02.825664 2321 tablet_replica.cc:330] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: stopping tablet replica
I20260809 05:22:02.825870 2321 raft_consensus.cc:2229] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:02.826078 2321 raft_consensus.cc:2258] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:02.827265 2321 ts_tablet_manager.cc:1905] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.831648 2320 ts_tablet_manager.cc:1918] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.831800 2320 log.cc:1196] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/27a7267d32e8454193a87c39dc095664
I20260809 05:22:02.832201 2320 ts_tablet_manager.cc:1939] T 27a7267d32e8454193a87c39dc095664 P 09a0db971ea4466eabbc49796a93cfe3: Deleting consensus metadata
I20260809 05:22:02.825086 2368 ts_tablet_manager.cc:1905] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:02.837356 1468 catalog_manager.cc:4841] TS 09a0db971ea4466eabbc49796a93cfe3 (127.22.1.66:44113): tablet 27a7267d32e8454193a87c39dc095664 (table test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]) successfully deleted
I20260809 05:22:02.838820 2321 ts_tablet_manager.cc:1918] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.839136 2321 log.cc:1196] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/27a7267d32e8454193a87c39dc095664
I20260809 05:22:02.839253 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 1600
I20260809 05:22:02.839737 2321 ts_tablet_manager.cc:1939] T 27a7267d32e8454193a87c39dc095664 P 33f4f98f54ce453eb523f6384cb40062: Deleting consensus metadata
I20260809 05:22:02.841038 1462 catalog_manager.cc:4841] TS 33f4f98f54ce453eb523f6384cb40062 (127.22.1.65:37757): tablet 27a7267d32e8454193a87c39dc095664 (table test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]) successfully deleted
I20260809 05:22:02.845556 2368 ts_tablet_manager.cc:1918] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260809 05:22:02.845824 2368 log.cc:1196] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/27a7267d32e8454193a87c39dc095664
I20260809 05:22:02.846350 2368 ts_tablet_manager.cc:1939] T 27a7267d32e8454193a87c39dc095664 P 3943e990350a40ac833ccbc92b3eea39: Deleting consensus metadata
I20260809 05:22:02.848078 1469 catalog_manager.cc:4841] TS 3943e990350a40ac833ccbc92b3eea39 (127.22.1.67:40427): tablet 27a7267d32e8454193a87c39dc095664 (table test-table [id=5f6a2a8f83bd45d8a88270592ec4c1d8]) successfully deleted
Aug 09 05:22:02 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.65@KRBTEST.COM
I20260809 05:22:02.863263 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:02.841553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.22.1.65:37757)
0809 05:22:02.841777 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:02.841783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:02.848420 (+ 6637us) server_negotiation.cc:407] Connection header received
0809 05:22:02.848721 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:02.848728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:02.848822 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:02.848907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:02.850369 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.851243 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.852639 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:02.853106 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:02.853282 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:02.857317 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:02.857335 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:02.857353 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:02.857390 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:02.860323 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:02.861000 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:02.861011 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:02.861013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:02.861203 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:02.861619 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:02.861622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:02.861625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:02.862012 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:02.862343 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:02.862662 (+ 319us) server_negotiation.cc:299] Negotiation successful
0809 05:22:02.862916 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260809 05:22:02.863942 1813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:03.025695 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 1600
I20260809 05:22:03.060108 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 1828
Aug 09 05:22:03 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.66@KRBTEST.COM
I20260809 05:22:03.076601 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:03.060925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.22.1.66:44113)
0809 05:22:03.061482 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:03.061487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:03.061502 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:22:03.061557 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:03.061560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:03.061626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:03.061717 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:03.062980 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:03.063724 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:03.064984 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:03.065464 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:03.065581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:03.070427 (+ 4846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:03.070445 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:03.070463 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:03.070539 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:03.073561 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:03.074457 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:03.074465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:03.074467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:03.074658 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:03.075076 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:03.075080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:03.075082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:03.075482 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:03.075774 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:03.076095 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 05:22:03.076277 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":271,"threads_started":1}
I20260809 05:22:03.077353 2014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:03.200026 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 1828
I20260809 05:22:03.223558 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 2028
Aug 09 05:22:03 dist-test-slave-rmhg krb5kdc[1412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.1.67@KRBTEST.COM
I20260809 05:22:03.238008 2394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:03.224337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.22.1.67:40427)
0809 05:22:03.224507 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:03.224512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:03.224553 (+ 41us) server_negotiation.cc:407] Connection header received
0809 05:22:03.224694 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:03.224697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:03.224785 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:03.224872 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:03.226206 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:03.226913 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:03.228135 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:03.228529 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:03.228624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:03.232834 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:03.232850 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:03.232868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:03.232905 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:03.235606 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:03.236207 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:03.236214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:03.236216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:03.236356 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:03.236658 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:03.236660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:03.236662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:03.237003 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:03.237261 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:03.237572 (+ 311us) server_negotiation.cc:299] Negotiation successful
0809 05:22:03.237744 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:22:03.238688 2230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:03.363948 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 2028
I20260809 05:22:03.391625 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 1437
I20260809 05:22:03.392880 1549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:03.508442 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 1437
2026-08-09T05:22:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5668 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:22:03 dist-test-slave-rmhg krb5kdc[2431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:22:03 dist-test-slave-rmhg krb5kdc[2431](info): set up 2 sockets
Aug 09 05:22:03 dist-test-slave-rmhg krb5kdc[2431](info): commencing operation
krb5kdc: starting...
W20260809 05:22:04.718984 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:22:04 dist-test-slave-rmhg krb5kdc[2431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:22:04.978986 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36061
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:44873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:05.323083 2459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:05.323424 2459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:05.323577 2459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:05.331200 2459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:05.331367 2459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:05.331451 2459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:05.331527 2459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:05.331604 2459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:05.342164 2459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:36061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:05.343685 2459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:05.345083 2459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:05.357438 2464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:05.357410 2466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:05.357365 2470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:05.358320 2459 server_base.cc:1034] running on GCE node
I20260809 05:22:05.359182 2459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:05.360771 2459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:05.362039 2459 hybrid_clock.cc:648] HybridClock initialized: now 1786252925361973 us; error 51 us; skew 500 ppm
I20260809 05:22:05.362427 2459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-09T05:22:05Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1914 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:22:05 dist-test-slave-rmhg krb5kdc[2482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:22:05 dist-test-slave-rmhg krb5kdc[2482](info): set up 2 sockets
Aug 09 05:22:05 dist-test-slave-rmhg krb5kdc[2482](info): commencing operation
krb5kdc: starting...
W20260809 05:22:06.633744 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:22:06 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:22:06.888906 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41931
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:07.225423 2504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:07.225731 2504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:07.225864 2504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:07.233186 2504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:07.233304 2504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:07.233379 2504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:07.233456 2504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:07.233516 2504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:07.243654 2504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:07.245096 2504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:07.246517 2504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:07.258232 2510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:07.258355 2511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:07.258282 2513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:07.259265 2504 server_base.cc:1034] running on GCE node
I20260809 05:22:07.260063 2504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:07.261623 2504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:07.262826 2504 hybrid_clock.cc:648] HybridClock initialized: now 1786252927262782 us; error 36 us; skew 500 ppm
I20260809 05:22:07.263188 2504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:07 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252927, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:07.269058 2504 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:22:07.270696 2504 webserver.cc:460] Webserver started at http://127.22.1.126:34161/ using document root <none> and password file <none>
I20260809 05:22:07.271404 2504 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:07.271521 2504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:07.271823 2504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:07.274950 2504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "190455fe5eafc7ce63b352c97f4ae2ce"
server_key_iv: "ba73ee932f90c55dfd2ee5cb16ceece8"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.275718 2504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "190455fe5eafc7ce63b352c97f4ae2ce"
server_key_iv: "ba73ee932f90c55dfd2ee5cb16ceece8"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.280844 2504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:22:07.284622 2524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:07.286021 2504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260809 05:22:07.286198 2504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "190455fe5eafc7ce63b352c97f4ae2ce"
server_key_iv: "ba73ee932f90c55dfd2ee5cb16ceece8"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.286383 2504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:07.302147 2504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:07.306262 2504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:07.306537 2504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:07.326061 2504 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41931
I20260809 05:22:07.326083 2603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41931 every 8 connection(s)
I20260809 05:22:07.327641 2504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:22:07.331693 2604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:07.334762 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 2504
I20260809 05:22:07.335019 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:22:07.335836 22533 external_mini_cluster.cc:1428] Setting key 332e7fd47485ede4499978e35560c8e4
I20260809 05:22:07.344223 2604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Bootstrap starting.
I20260809 05:22:07.347543 2604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:07.348644 2604 log.cc:824] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:22:07 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:07.351616 2604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: No bootstrap required, opened a new log
I20260809 05:22:07.358331 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:07.338160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.22.1.126:41931)
0809 05:22:07.338970 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:07.338982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:07.339024 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:22:07.340732 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:07.340756 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:07.341229 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:07.341647 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:07.343145 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.344531 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.345956 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.346652 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.346828 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:07.350723 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:07.350731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:07.350747 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:07.350789 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:07.354026 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:07.354754 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:07.354762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:07.354768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:07.354951 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:07.355356 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:07.355360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:07.355362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:07.355959 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:07.356280 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:07.356762 (+ 482us) server_negotiation.cc:299] Negotiation successful
0809 05:22:07.357241 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":359,"threads_started":1}
I20260809 05:22:07.359609 2604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:07.359975 2604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:07.360075 2604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfb8c5ef98b4432a8ce18db80c0a9e6, State: Initialized, Role: FOLLOWER
I20260809 05:22:07.360627 2604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:07.360849 2604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:22:07.361032 2604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:22:07.361299 2604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:07.363510 2604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:07.364104 2604 leader_election.cc:302] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dfb8c5ef98b4432a8ce18db80c0a9e6; no voters:
I20260809 05:22:07.364763 2604 leader_election.cc:288] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:22:07.364867 2612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:07.365566 2612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 4dfb8c5ef98b4432a8ce18db80c0a9e6, State: Running, Role: LEADER
I20260809 05:22:07.366168 2612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:07.367136 2604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:22:07.369102 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dfb8c5ef98b4432a8ce18db80c0a9e6. Latest consensus state: current_term: 1 leader_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } } }
I20260809 05:22:07.369508 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:07.370025 2614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } } }
I20260809 05:22:07.370342 2614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:07.371027 2624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:22:07.376381 2624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:22:07.386747 2624 catalog_manager.cc:1347] Generated new cluster ID: a3f0a49387a943cb9534d8d0e569103c
I20260809 05:22:07.386884 2624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:22:07.408758 2624 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:22:07.410187 2624 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:22:07.418043 2624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Generated new TSK 0
I20260809 05:22:07.418975 2624 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:22:07.494539 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:07.837311 2648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:07.837617 2648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:07.837747 2648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:07.845072 2648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:07.845237 2648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:07.845405 2648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:07.855741 2648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:07.857134 2648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:07.858644 2648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:07.872028 2655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:07.872107 2659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:07.872073 2656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:07.873867 2648 server_base.cc:1034] running on GCE node
I20260809 05:22:07.874539 2648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:07.875615 2648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:07.876828 2648 hybrid_clock.cc:648] HybridClock initialized: now 1786252927876763 us; error 58 us; skew 500 ppm
I20260809 05:22:07.877190 2648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:07 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252927, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:07.880374 2648 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:22:07.881982 2648 webserver.cc:460] Webserver started at http://127.22.1.65:38651/ using document root <none> and password file <none>
I20260809 05:22:07.882704 2648 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:07.882826 2648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:07.883144 2648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:07.886229 2648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "407922e1776e46dd8c202fb0fbad9050"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "502231de83c1e46838cb0309ee90acf5"
server_key_iv: "1e9f1c0248fd11f54ce3a34470710a6d"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.887061 2648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "407922e1776e46dd8c202fb0fbad9050"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "502231de83c1e46838cb0309ee90acf5"
server_key_iv: "1e9f1c0248fd11f54ce3a34470710a6d"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.892444 2648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:22:07.896142 2671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:07.897624 2648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:07.897779 2648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "407922e1776e46dd8c202fb0fbad9050"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "502231de83c1e46838cb0309ee90acf5"
server_key_iv: "1e9f1c0248fd11f54ce3a34470710a6d"
server_key_version: "encryptionkey@0"
I20260809 05:22:07.897969 2648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:07.916970 2648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:07.921551 2648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:07.921868 2648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:07.922855 2648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:07.924342 2648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:07.924443 2648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:07.924573 2648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:07.924630 2648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:07.956162 2648 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:38353
I20260809 05:22:07.956213 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:38353 every 8 connection(s)
I20260809 05:22:07.957573 2648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:07.960765 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 2648
I20260809 05:22:07.961005 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:22:07.961896 22533 external_mini_cluster.cc:1428] Setting key 7a081bf4a9ebce4212e12923c4ba86df
Aug 09 05:22:07 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252927, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:07.978845 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:07.960192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:42709 (local address 127.22.1.126:41931)
0809 05:22:07.960642 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:07.960647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:07.961867 (+ 1220us) server_negotiation.cc:407] Connection header received
0809 05:22:07.963947 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:07.963953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:07.964037 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:07.964183 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:07.965837 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.966839 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.968620 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.969195 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.969317 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:07.972957 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:07.972964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:07.972969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:07.973012 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:07.975539 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:07.976459 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:07.976463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:07.976465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:07.976593 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:07.977067 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:07.977070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:07.977072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:07.977368 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:07.977479 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:07.978399 (+ 920us) server_negotiation.cc:299] Negotiation successful
0809 05:22:07.978593 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260809 05:22:07.980361 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:07.960811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:42709)
0809 05:22:07.961688 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:07.961742 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:07.963639 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:07.964405 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:07.964414 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:07.965005 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:07.965661 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:07.965680 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.967007 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.967011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:07.968417 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:07.968462 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:07.969412 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:07.969418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:07.969658 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:07.970350 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:07.970379 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:07.972703 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:07.975689 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:07.975698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:07.975717 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:07.976344 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:07.976696 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:07.976700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:07.976703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:07.976968 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:07.977487 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:07.977495 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:07.978008 (+ 513us) client_negotiation.cc:769] Sending connection context
0809 05:22:07.978328 (+ 320us) client_negotiation.cc:240] Negotiation successful
0809 05:22:07.978892 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":369,"threads_started":1}
I20260809 05:22:07.982064 2857 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41931
I20260809 05:22:07.982421 2857 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:07.983410 2857 heartbeater.cc:511] Master 127.22.1.126:41931 requested a full tablet report, sending...
I20260809 05:22:07.986014 2553 ts_manager.cc:194] Registered new tserver with Master: 407922e1776e46dd8c202fb0fbad9050 (127.22.1.65:38353)
I20260809 05:22:07.987617 2553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:42709
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:22:08.020462 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:08.367074 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:08.367389 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:08.367506 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:08.375056 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:08.375196 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:08.375352 2864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:08.387243 2864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:08.388671 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:08.390091 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:08.403646 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:08.403676 2880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:08.403712 2877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:08.404498 2864 server_base.cc:1034] running on GCE node
I20260809 05:22:08.405125 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:08.406286 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:08.407509 2864 hybrid_clock.cc:648] HybridClock initialized: now 1786252928407457 us; error 45 us; skew 500 ppm
I20260809 05:22:08.407819 2864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:08 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:08.411176 2864 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:22:08.412796 2864 webserver.cc:460] Webserver started at http://127.22.1.66:33483/ using document root <none> and password file <none>
I20260809 05:22:08.413566 2864 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:08.413677 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:08.413964 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:08.417102 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "396d613370b74ce7b5ee5d44894dff1a"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "726e106d0f8cb0924b4314d4d1185f01"
server_key_iv: "a086acf21257fbed61988e59d16849cb"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.417928 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "396d613370b74ce7b5ee5d44894dff1a"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "726e106d0f8cb0924b4314d4d1185f01"
server_key_iv: "a086acf21257fbed61988e59d16849cb"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.423236 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:22:08.427094 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.428520 2864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:22:08.428680 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "396d613370b74ce7b5ee5d44894dff1a"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "726e106d0f8cb0924b4314d4d1185f01"
server_key_iv: "a086acf21257fbed61988e59d16849cb"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.428889 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:08.462777 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:08.467542 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:08.467829 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:08.468907 2864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:08.470355 2864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:08.470431 2864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.470542 2864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:08.470587 2864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.501821 2864 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:45201
I20260809 05:22:08.501856 3058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:45201 every 8 connection(s)
I20260809 05:22:08.503134 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:08.507417 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 2864
I20260809 05:22:08.507675 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:22:08.508237 22533 external_mini_cluster.cc:1428] Setting key 58443a4725a69ab861693efefb32752b
Aug 09 05:22:08 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:08.525674 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:08.505790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:47247 (local address 127.22.1.126:41931)
0809 05:22:08.506207 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:08.506212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:08.507364 (+ 1152us) server_negotiation.cc:407] Connection header received
0809 05:22:08.509720 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:08.509726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:08.509804 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:08.509924 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:08.511962 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:08.512714 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:08.514949 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:08.515378 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:08.515486 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:08.519703 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:08.519708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:08.519712 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:08.519747 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:08.522341 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:08.523237 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:08.523241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:08.523243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:08.523375 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:08.523860 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:08.523863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:08.523865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:08.524135 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:08.524242 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:08.525142 (+ 900us) server_negotiation.cc:299] Negotiation successful
0809 05:22:08.525395 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":200,"threads_started":1}
I20260809 05:22:08.527148 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:08.506374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:47247)
0809 05:22:08.507136 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:08.507199 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:08.509439 (+ 2240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:08.510150 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:08.510159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:08.510875 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:08.511751 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:08.511781 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:08.512890 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:08.512895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:08.514782 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:08.514799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:08.515624 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:08.515631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:08.515910 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:08.516681 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:08.516717 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:08.519443 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:08.522503 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:08.522512 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:08.522533 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:08.523118 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:08.523480 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:08.523485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:08.523488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:08.523765 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:08.524252 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:08.524260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:08.524716 (+ 456us) client_negotiation.cc:769] Sending connection context
0809 05:22:08.525080 (+ 364us) client_negotiation.cc:240] Negotiation successful
0809 05:22:08.525670 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":282,"threads_started":1}
I20260809 05:22:08.528769 3062 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41931
I20260809 05:22:08.529093 3062 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:08.529862 3062 heartbeater.cc:511] Master 127.22.1.126:41931 requested a full tablet report, sending...
I20260809 05:22:08.531409 2553 ts_manager.cc:194] Registered new tserver with Master: 396d613370b74ce7b5ee5d44894dff1a (127.22.1.66:45201)
I20260809 05:22:08.532567 2553 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:47247
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:22:08.568087 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:08.903087 3073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:08.903367 3073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:08.903460 3073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:08.910696 3073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:08.910822 3073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:08.910936 3073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:08.920889 3073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:08.922231 3073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:08.923537 3073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:08.937004 3087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:08.936960 3090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:08.937041 3085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:08.938069 3073 server_base.cc:1034] running on GCE node
I20260809 05:22:08.938709 3073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:08.939654 3073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:08.940853 3073 hybrid_clock.cc:648] HybridClock initialized: now 1786252928940807 us; error 35 us; skew 500 ppm
I20260809 05:22:08.941170 3073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:08 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:08.944278 3073 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:22:08.945832 3073 webserver.cc:460] Webserver started at http://127.22.1.67:36209/ using document root <none> and password file <none>
I20260809 05:22:08.946493 3073 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:08.946599 3073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:08.946868 3073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:08.949782 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "41ea90cd604349dbaec0b983849de22d"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "445e860a62fffca8a7930a80afd45efd"
server_key_iv: "1a1eda722ae4ae452dd5839729504600"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.950502 3073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "41ea90cd604349dbaec0b983849de22d"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "445e860a62fffca8a7930a80afd45efd"
server_key_iv: "1a1eda722ae4ae452dd5839729504600"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.955930 3073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:22:08.960367 3098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.961982 3073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:08.962121 3073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "41ea90cd604349dbaec0b983849de22d"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "445e860a62fffca8a7930a80afd45efd"
server_key_iv: "1a1eda722ae4ae452dd5839729504600"
server_key_version: "encryptionkey@0"
I20260809 05:22:08.962294 3073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:08.974990 3073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:08.979265 3073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:08.979542 3073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:08.980492 3073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:08.981989 3073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:08.982065 3073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.982165 3073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:08.982218 3073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:08.990964 2857 heartbeater.cc:503] Master 127.22.1.126:41931 was elected leader, sending a full tablet report...
I20260809 05:22:09.013351 3073 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:37079
I20260809 05:22:09.013434 3266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:37079 every 8 connection(s)
I20260809 05:22:09.014667 3073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:09.023931 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 3073
I20260809 05:22:09.024163 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:22:09.024755 22533 external_mini_cluster.cc:1428] Setting key 6e74ac2048d5d6828db920aa85fe74d7
Aug 09 05:22:09 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252928, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:09.035163 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:09.017372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:33621 (local address 127.22.1.126:41931)
0809 05:22:09.017595 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:09.017601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:09.018846 (+ 1245us) server_negotiation.cc:407] Connection header received
0809 05:22:09.020692 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:09.020698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:09.020795 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:09.020885 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:09.022489 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.023232 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.025518 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.026035 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.026257 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:09.030010 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:09.030015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:09.030018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:09.030053 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:09.032398 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.033136 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.033142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.033145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.033300 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.033690 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.033694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.033696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.034000 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:09.034117 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:09.034734 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 05:22:09.034906 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 05:22:09.036095 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:09.017866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:33621)
0809 05:22:09.018653 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:09.018702 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:09.020400 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:09.021136 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:09.021144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:09.021734 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:09.022310 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:09.022328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.023385 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.023389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:09.025379 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:09.025392 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.026706 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.026716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:09.026927 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:09.027558 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:09.027611 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:09.029799 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:09.032544 (+ 2745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:09.032552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:09.032567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:09.032983 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:09.033390 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:09.033393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:09.033395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:09.033579 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:09.034101 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:09.034107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:09.034421 (+ 314us) client_negotiation.cc:769] Sending connection context
0809 05:22:09.034643 (+ 222us) client_negotiation.cc:240] Negotiation successful
0809 05:22:09.035025 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":289,"threads_started":1}
I20260809 05:22:09.037463 3267 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41931
I20260809 05:22:09.037742 3267 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:09.038462 3267 heartbeater.cc:511] Master 127.22.1.126:41931 requested a full tablet report, sending...
I20260809 05:22:09.039995 2551 ts_manager.cc:194] Registered new tserver with Master: 41ea90cd604349dbaec0b983849de22d (127.22.1.67:37079)
I20260809 05:22:09.040874 2551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:33621
I20260809 05:22:09.052394 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
Entry for principal oryx/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab.
WARNING: no policy specified for oryx/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
Entry for principal oryx/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab.
WARNING: no policy specified for oryx/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
Entry for principal oryx/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab.
I20260809 05:22:09.202250 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 2648
Aug 09 05:22:09 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:22:09.218010 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:09.203362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.22.1.65:38353)
0809 05:22:09.203994 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:09.204001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:09.204029 (+ 28us) server_negotiation.cc:407] Connection header received
0809 05:22:09.204092 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:09.204098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:09.204312 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:09.204467 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:09.205957 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.206969 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.208179 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.208765 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.208864 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:09.212741 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:09.212753 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:09.212770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:09.212803 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:09.215427 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.216124 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.216131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.216134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.216264 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.216623 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.216626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.216628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.216961 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:09.217223 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:09.217490 (+ 267us) server_negotiation.cc:299] Negotiation successful
0809 05:22:09.217693 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":327,"threads_started":1}
I20260809 05:22:09.219287 2848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:09.301061 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 2648
I20260809 05:22:09.323848 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 2864
Aug 09 05:22:09 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:22:09.343089 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:09.324942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.22.1.66:45201)
0809 05:22:09.325621 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:09.325630 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:09.325663 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:22:09.325743 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:09.325752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:09.326042 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:09.326211 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:09.327652 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.329429 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.331264 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.332301 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.332438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:09.337359 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:09.337373 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:09.337391 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:09.337432 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:09.340317 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.341031 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.341039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.341041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.341223 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.341545 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.341548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.341550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.341911 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:09.342139 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:09.342476 (+ 337us) server_negotiation.cc:299] Negotiation successful
0809 05:22:09.342700 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":333,"threads_started":1}
I20260809 05:22:09.344377 3053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:09.426808 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 2864
I20260809 05:22:09.444936 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3073
Aug 09 05:22:09 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:22:09.461591 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:09.446041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.22.1.67:37079)
0809 05:22:09.446630 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:09.446637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:09.446666 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:22:09.446726 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:09.446732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:09.446940 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:09.447086 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:09.448440 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.449699 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.450990 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:09.451562 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:09.451677 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:09.455934 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:09.455947 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:09.455965 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:09.456000 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:09.458752 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.459422 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.459430 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.459432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.459588 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:09.459952 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:09.459956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:09.459958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:09.460324 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:09.460681 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:09.460937 (+ 256us) server_negotiation.cc:299] Negotiation successful
0809 05:22:09.461239 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":289,"threads_started":1}
I20260809 05:22:09.462944 3257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:09.545113 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 3073
I20260809 05:22:09.566275 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 2504
I20260809 05:22:09.567586 2596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:09.670178 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 2504
I20260809 05:22:09.696877 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41931
--webserver_interface=127.22.1.126
--webserver_port=34161
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:10.048554 3301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:10.048887 3301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:10.049005 3301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:10.056370 3301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:10.056485 3301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:10.056538 3301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:10.056619 3301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:10.056679 3301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:10.066722 3301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=34161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:10.068095 3301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:10.069475 3301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:10.081550 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:10.081588 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:10.081514 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:10.082309 3301 server_base.cc:1034] running on GCE node
I20260809 05:22:10.083081 3301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:10.084544 3301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:10.085750 3301 hybrid_clock.cc:648] HybridClock initialized: now 1786252930085703 us; error 31 us; skew 500 ppm
I20260809 05:22:10.086087 3301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252930, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:10.089203 3301 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:22:10.090773 3301 webserver.cc:460] Webserver started at http://127.22.1.126:34161/ using document root <none> and password file <none>
I20260809 05:22:10.091436 3301 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:10.091553 3301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:10.097010 3301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:10.099787 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:10.101243 3301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:10.101393 3301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "190455fe5eafc7ce63b352c97f4ae2ce"
server_key_iv: "ba73ee932f90c55dfd2ee5cb16ceece8"
server_key_version: "encryptionkey@0"
I20260809 05:22:10.102344 3301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:10.118558 3301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:10.123049 3301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:10.123292 3301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:10.141501 3301 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41931
I20260809 05:22:10.141527 3397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41931 every 8 connection(s)
I20260809 05:22:10.143270 3301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:22:10.150393 3398 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:22:10.151821 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 3301
I20260809 05:22:10.152607 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:38353
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:41931
--webserver_port=38651
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:22:10.153856 3398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Bootstrap starting.
I20260809 05:22:10.172851 3398 log.cc:824] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:10.182185 3398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:22:10.182704 3398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Bootstrap complete.
I20260809 05:22:10.191792 3398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:10.192656 3398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfb8c5ef98b4432a8ce18db80c0a9e6, State: Initialized, Role: FOLLOWER
I20260809 05:22:10.193342 3398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:10.193576 3398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:22:10.193684 3398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:22:10.193845 3398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:22:10.197669 3398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:10.198160 3398 leader_election.cc:302] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dfb8c5ef98b4432a8ce18db80c0a9e6; no voters:
I20260809 05:22:10.198779 3398 leader_election.cc:288] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 05:22:10.198926 3405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 05:22:10.200167 3405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [term 2 LEADER]: Becoming Leader. State: Replica: 4dfb8c5ef98b4432a8ce18db80c0a9e6, State: Running, Role: LEADER
I20260809 05:22:10.201000 3398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:22:10.200875 3405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } }
I20260809 05:22:10.203485 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } } }
I20260809 05:22:10.203871 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:10.206176 3407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dfb8c5ef98b4432a8ce18db80c0a9e6. Latest consensus state: current_term: 2 leader_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dfb8c5ef98b4432a8ce18db80c0a9e6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41931 } } }
I20260809 05:22:10.206521 3407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:10.207813 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:22:10.213703 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:22:10.215214 3411 catalog_manager.cc:1259] Loaded cluster ID: a3f0a49387a943cb9534d8d0e569103c
I20260809 05:22:10.215380 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:22:10.219398 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:22:10.221212 3411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4dfb8c5ef98b4432a8ce18db80c0a9e6: Loaded TSK: 0
I20260809 05:22:10.222864 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 05:22:10.506608 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:10.506946 3400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:10.507072 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:10.514515 3400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:10.514637 3400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:10.514781 3400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:10.525012 3400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.65.keytab
--principal=oryx/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:38353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=38651
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:10.526587 3400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:10.527987 3400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:10.543339 3448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:10.543370 3450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:10.544094 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:10.544286 3400 server_base.cc:1034] running on GCE node
I20260809 05:22:10.545008 3400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:10.546070 3400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:10.547294 3400 hybrid_clock.cc:648] HybridClock initialized: now 1786252930547233 us; error 48 us; skew 500 ppm
I20260809 05:22:10.547657 3400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252930, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:10.551154 3400 init.cc:377] Logged in from keytab as oryx/127.22.1.65@KRBTEST.COM (short username oryx)
I20260809 05:22:10.552964 3400 webserver.cc:460] Webserver started at http://127.22.1.65:38651/ using document root <none> and password file <none>
I20260809 05:22:10.553871 3400 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:10.554036 3400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:10.560990 3400 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.005s
I20260809 05:22:10.564077 3461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:10.565508 3400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:10.565656 3400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "407922e1776e46dd8c202fb0fbad9050"
format_stamp: "Formatted at 2026-08-09 05:22:07 on dist-test-slave-rmhg"
server_key: "502231de83c1e46838cb0309ee90acf5"
server_key_iv: "1e9f1c0248fd11f54ce3a34470710a6d"
server_key_version: "encryptionkey@0"
I20260809 05:22:10.566582 3400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:10.586064 3400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:10.590348 3400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:10.590602 3400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:10.591603 3400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:10.593041 3400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:10.593130 3400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:10.593253 3400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:10.593318 3400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:10.624401 3400 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:38353
I20260809 05:22:10.624471 3649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:38353 every 8 connection(s)
I20260809 05:22:10.625865 3400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:10.628847 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 3400
I20260809 05:22:10.629632 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:45201
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:41931
--webserver_port=33483
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 05:22:10.651173 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:10.629011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:40405 (local address 127.22.1.126:41931)
0809 05:22:10.630179 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:10.630191 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:10.633912 (+ 3721us) server_negotiation.cc:407] Connection header received
0809 05:22:10.636443 (+ 2531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:10.636462 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:10.636866 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:10.637258 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:10.638854 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.640684 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.643012 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.643734 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.643895 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.650251 (+ 6356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:40405: BlockingRecv error: recv got EOF from 127.22.1.65:40405 (error 108)
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":303,"threads_started":1}
I20260809 05:22:10.651214 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:10.629279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:40405)
0809 05:22:10.633709 (+ 4430us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:10.633751 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:10.635553 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:10.637470 (+ 1917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:10.637482 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:10.638099 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:10.638653 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.638672 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.640855 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.640862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.642817 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.642850 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.644012 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.644020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.644319 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.645183 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:10.645225 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:10.649825 (+ 4600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4142,"thread_start_us":3962,"threads_started":1}
W20260809 05:22:10.652118 3650 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41931 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.1.126:41931: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:10.661322 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:10.652792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:44457)
0809 05:22:10.653000 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:10.653020 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:10.653124 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:10.653507 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:10.653510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:10.653822 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:10.653978 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.653985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.655429 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.655435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.656352 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.656365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.657292 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.657296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.657427 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.658023 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:10.658042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:10.660994 (+ 2952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 05:22:10.661465 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:10.652960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:44457 (local address 127.22.1.126:41931)
0809 05:22:10.653199 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:10.653205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:10.653225 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:22:10.653292 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:10.653298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:10.653379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:10.653489 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:10.654118 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.655259 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.656504 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.657103 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.657277 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.661250 (+ 3973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:44457: BlockingRecv error: recv got EOF from 127.22.1.65:44457 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:10 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:10.669873 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:10.662196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:52361)
0809 05:22:10.662400 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:10.662419 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:10.662568 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:10.662877 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:10.662881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:10.663192 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:10.663340 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.663348 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.664332 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.664336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.665312 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:10.665324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.665977 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.665982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:10.666175 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.666704 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:10.666722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:10.669566 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 05:22:10.670212 3650 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 05:22:10.670375 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:10.662269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:52361 (local address 127.22.1.126:41931)
0809 05:22:10.662432 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:10.662438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:10.662525 (+ 87us) server_negotiation.cc:407] Connection header received
0809 05:22:10.662675 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:10.662680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:10.662752 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:10.662835 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:10.663477 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.664189 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.665453 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:10.665838 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:10.665960 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:10.669820 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:52361: BlockingRecv error: recv got EOF from 127.22.1.65:52361 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 05:22:10.968961 3655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:10.969313 3655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:10.969468 3655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:10.976895 3655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:10.977026 3655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:10.977210 3655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:10.987250 3655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.66.keytab
--principal=oryx/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:45201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=33483
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:10.988725 3655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:10.990166 3655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:11.003471 3663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:11.003525 3666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:11.003432 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:11.004204 3655 server_base.cc:1034] running on GCE node
I20260809 05:22:11.004860 3655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:11.005978 3655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:11.007181 3655 hybrid_clock.cc:648] HybridClock initialized: now 1786252931007120 us; error 57 us; skew 500 ppm
I20260809 05:22:11.007472 3655 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252931, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:11.010733 3655 init.cc:377] Logged in from keytab as oryx/127.22.1.66@KRBTEST.COM (short username oryx)
I20260809 05:22:11.012153 3655 webserver.cc:460] Webserver started at http://127.22.1.66:33483/ using document root <none> and password file <none>
I20260809 05:22:11.012843 3655 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:11.012955 3655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:11.018477 3655 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260809 05:22:11.021364 3678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.022850 3655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:11.023010 3655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "396d613370b74ce7b5ee5d44894dff1a"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "726e106d0f8cb0924b4314d4d1185f01"
server_key_iv: "a086acf21257fbed61988e59d16849cb"
server_key_version: "encryptionkey@0"
I20260809 05:22:11.024019 3655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:11.046808 3655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:11.051431 3655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:11.051700 3655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:11.052750 3655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:11.054173 3655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:11.054261 3655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.054366 3655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:11.054431 3655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.084513 3655 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:45201
I20260809 05:22:11.084589 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:45201 every 8 connection(s)
I20260809 05:22:11.086030 3655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:11.091385 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 3655
I20260809 05:22:11.091979 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:37079
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:41931
--webserver_port=36209
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 05:22:11.106174 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.088714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:45597 (local address 127.22.1.126:41931)
0809 05:22:11.088904 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.088909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.090197 (+ 1288us) server_negotiation.cc:407] Connection header received
0809 05:22:11.092154 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.092161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.092251 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.092359 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.093979 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.094911 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.096525 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.097078 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.097230 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.105917 (+ 8687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:45597: BlockingRecv error: recv got EOF from 127.22.1.66:45597 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:22:11.107136 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.089276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:45597)
0809 05:22:11.090016 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.090060 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.091844 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.092609 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.092617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.093168 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.093781 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.093801 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.095093 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.095097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.096374 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.096388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.097342 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.097347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.097563 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.101218 (+ 3655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.101259 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.105745 (+ 4486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":291,"threads_started":1}
W20260809 05:22:11.107925 3852 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41931 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.1.126:41931: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:11.116895 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.108578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:55037)
0809 05:22:11.108749 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.108769 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.108865 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.109252 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.109255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.109636 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.109794 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.109806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.111113 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.111117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.112017 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.112028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.112885 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.112889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.113069 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.113651 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.113669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.116603 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 05:22:11.116999 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.108658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:55037 (local address 127.22.1.126:41931)
0809 05:22:11.108826 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.108832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.108854 (+ 22us) server_negotiation.cc:407] Connection header received
0809 05:22:11.108987 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.108992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.109078 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.109200 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.109932 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.110958 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.112156 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.112727 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.112858 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.116805 (+ 3947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:55037: BlockingRecv error: recv got EOF from 127.22.1.66:55037 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:11.126485 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.117682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:59009)
0809 05:22:11.117856 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.117876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.117986 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.118308 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.118313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.118696 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.118900 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.118914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.119902 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.119908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.121429 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.121445 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.122190 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.122196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.122375 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.122915 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.122936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.126184 (+ 3248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 05:22:11.126562 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.117822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:59009 (local address 127.22.1.126:41931)
0809 05:22:11.117970 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.117982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.117999 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:22:11.118110 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.118113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.118173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.118250 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.119060 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.119761 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.121564 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.122054 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.122150 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.126390 (+ 4240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:59009: BlockingRecv error: recv got EOF from 127.22.1.66:59009 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260809 05:22:11.126732 3852 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 05:22:11.457480 3855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:11.457791 3855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:11.457897 3855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:11.465580 3855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:11.465703 3855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:11.465823 3855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:11.475888 3855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/krb5kdc/oryx_127.22.1.67.keytab
--principal=oryx/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:37079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=36209
--tserver_master_addrs=127.22.1.126:41931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:11.477325 3855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:11.478708 3855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:11.491601 3862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:11.491750 3866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:11.491693 3864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:11.492892 3855 server_base.cc:1034] running on GCE node
I20260809 05:22:11.493718 3855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:11.494871 3855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:11.496119 3855 hybrid_clock.cc:648] HybridClock initialized: now 1786252931496059 us; error 48 us; skew 500 ppm
I20260809 05:22:11.496459 3855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252931, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:11.499723 3855 init.cc:377] Logged in from keytab as oryx/127.22.1.67@KRBTEST.COM (short username oryx)
I20260809 05:22:11.501413 3855 webserver.cc:460] Webserver started at http://127.22.1.67:36209/ using document root <none> and password file <none>
I20260809 05:22:11.502204 3855 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:11.502339 3855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:11.508363 3855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260809 05:22:11.511467 3878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.512995 3855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:22:11.513216 3855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "41ea90cd604349dbaec0b983849de22d"
format_stamp: "Formatted at 2026-08-09 05:22:08 on dist-test-slave-rmhg"
server_key: "445e860a62fffca8a7930a80afd45efd"
server_key_iv: "1a1eda722ae4ae452dd5839729504600"
server_key_version: "encryptionkey@0"
I20260809 05:22:11.514312 3855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:11.535593 3855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:11.540233 3855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:11.540513 3855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:11.541675 3855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:11.543243 3855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:11.543358 3855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.543467 3855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:11.543546 3855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:11.574949 3855 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:37079
I20260809 05:22:11.575011 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:37079 every 8 connection(s)
I20260809 05:22:11.576447 3855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:11.581990 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 3855
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
W20260809 05:22:11.594780 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.579207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:34733 (local address 127.22.1.126:41931)
0809 05:22:11.579425 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.579430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.580609 (+ 1179us) server_negotiation.cc:407] Connection header received
0809 05:22:11.582595 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.582601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.582700 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.582821 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.584988 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.586159 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.588247 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.588882 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.589049 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.594469 (+ 5420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:34733: BlockingRecv error: recv got EOF from 127.22.1.67:34733 (error 108)
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":86}
I20260809 05:22:11.595497 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.579659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:34733)
0809 05:22:11.580410 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.580455 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.582278 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.583196 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.583211 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.584038 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.584761 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.584781 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.586341 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.586347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.588094 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.588108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.589194 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.589204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.589517 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.590357 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.590387 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.594212 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":310,"threads_started":1}
W20260809 05:22:11.596251 4046 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:41931 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.1.126:41931: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
I20260809 05:22:11.599691 4051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.583040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.22.1.126:41931)
0809 05:22:11.583818 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.583824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.583843 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:22:11.583893 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.583898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.583967 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.584089 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.586126 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.587293 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.589000 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.589650 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.589786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.592559 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:11.592567 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:11.592590 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:11.592629 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:11.595777 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:11.596546 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:11.596555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:11.596564 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:11.596783 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:11.597301 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:11.597306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:11.597309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:11.598022 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:11.598351 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:11.598849 (+ 498us) server_negotiation.cc:299] Negotiation successful
0809 05:22:11.599262 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":662,"thread_start_us":532,"threads_started":1}
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:11.605473 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.597000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:39833)
0809 05:22:11.597238 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.597258 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.597378 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.597677 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.597681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.598097 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.598298 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.598310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.599675 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.599680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.600708 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.600723 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.601663 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.601668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.601801 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.602410 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.602429 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.605269 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 05:22:11.606050 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.597060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:39833 (local address 127.22.1.126:41931)
0809 05:22:11.597248 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.597254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.597334 (+ 80us) server_negotiation.cc:407] Connection header received
0809 05:22:11.597481 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.597484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.597544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.597620 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.598475 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.599525 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.600837 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.601508 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.601651 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.605449 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:39833: BlockingRecv error: recv got EOF from 127.22.1.67:39833 (error 108)
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":61}
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:11.613482 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.606152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:44427)
0809 05:22:11.606375 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.606389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.606467 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.606727 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.606730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.606995 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.607153 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.607161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.608132 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.608135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.609102 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.609113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.609875 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.609881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.610039 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.610543 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.610561 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.613256 (+ 2695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126}
W20260809 05:22:11.613662 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.606243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:44427 (local address 127.22.1.126:41931)
0809 05:22:11.606441 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.606444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.606460 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:22:11.606542 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.606545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.606622 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.606703 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.607258 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.608026 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.609247 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.609700 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.609795 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.613427 (+ 3632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:44427: BlockingRecv error: recv got EOF from 127.22.1.67:44427 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260809 05:22:11.613819 4046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:11 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:11.679822 4052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:11.670841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:41493)
0809 05:22:11.671367 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:11.671384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:11.671513 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:11.671820 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:11.671823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:11.672301 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:11.672493 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.672502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.673507 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.673512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.674483 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:11.674495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.675342 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.675348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:11.675519 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.676532 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:11.676558 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:11.679583 (+ 3025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":296,"threads_started":1}
W20260809 05:22:11.679991 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:11.670938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:41493 (local address 127.22.1.126:41931)
0809 05:22:11.671157 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:11.671164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:11.671468 (+ 304us) server_negotiation.cc:407] Connection header received
0809 05:22:11.671626 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:11.671630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:11.671697 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:11.671784 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:11.672623 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.673373 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.674686 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:11.675158 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:11.675264 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:11.679785 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:41493: BlockingRecv error: recv got EOF from 127.22.1.65:41493 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:12.136227 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:12.127364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:43735)
0809 05:22:12.127883 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:12.127904 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:12.128047 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:12.128460 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:12.128464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:12.129039 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:12.129290 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.129303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.130493 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.130499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.131699 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.131715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.132476 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.132483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.132618 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.133223 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:12.133240 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:12.135989 (+ 2749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":286,"threads_started":1}
W20260809 05:22:12.136452 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:12.127480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:43735 (local address 127.22.1.126:41931)
0809 05:22:12.127678 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:12.127685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:12.128000 (+ 315us) server_negotiation.cc:407] Connection header received
0809 05:22:12.128168 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:12.128174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:12.128256 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:12.128379 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:12.129435 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.130325 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.131872 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.132300 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.132410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.136213 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:43735: BlockingRecv error: recv got EOF from 127.22.1.66:43735 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:12.627218 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:12.614478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:37415)
0809 05:22:12.615030 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:12.615047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:12.615187 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:12.615518 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:12.615523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:12.616037 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:12.616235 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.616245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.617356 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.617361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.618415 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.618428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.619152 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.619158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.619319 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.623749 (+ 4430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:12.623771 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:12.626987 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
W20260809 05:22:12.627460 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:12.614657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:37415 (local address 127.22.1.126:41931)
0809 05:22:12.614906 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:12.614912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:12.615135 (+ 223us) server_negotiation.cc:407] Connection header received
0809 05:22:12.615309 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:12.615313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:12.615380 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:12.615466 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:12.616388 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.617207 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.618553 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.618959 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.619071 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.627236 (+ 8165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:37415: BlockingRecv error: recv got EOF from 127.22.1.67:37415 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:12 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:12.689532 4056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:12.681036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:46967)
0809 05:22:12.681605 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:12.681622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:12.681770 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:12.682169 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:12.682174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:12.682617 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:12.682804 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.682817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.684050 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.684054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.684979 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:12.684992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.685726 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.685730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:12.685840 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.686441 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:12.686459 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:12.689286 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
W20260809 05:22:12.689692 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:12.681097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:46967 (local address 127.22.1.126:41931)
0809 05:22:12.681280 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:12.681284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:12.681720 (+ 436us) server_negotiation.cc:407] Connection header received
0809 05:22:12.681892 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:12.681896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:12.681983 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:12.682109 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:12.682976 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.683900 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.685136 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:12.685599 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:12.685708 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:12.689466 (+ 3758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:46967: BlockingRecv error: recv got EOF from 127.22.1.65:46967 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:13.145742 4058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:13.137231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:40163)
0809 05:22:13.137729 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:13.137749 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:13.137867 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:13.138144 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:13.138147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:13.138603 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:13.138750 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.138759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.139720 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.139725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.140766 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.140783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.141523 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.141529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.141684 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.142253 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:13.142270 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:13.145427 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":275,"threads_started":1}
W20260809 05:22:13.145888 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:13.137331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:40163 (local address 127.22.1.126:41931)
0809 05:22:13.137554 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:13.137560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:13.137857 (+ 297us) server_negotiation.cc:407] Connection header received
0809 05:22:13.137963 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:13.137967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:13.138032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:13.138116 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:13.138857 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.139571 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.140895 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.141372 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.141493 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.145663 (+ 4170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:40163: BlockingRecv error: recv got EOF from 127.22.1.66:40163 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:13.636747 4059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:13.628256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:53573)
0809 05:22:13.628758 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:13.628775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:13.628904 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:13.629239 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:13.629243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:13.629686 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:13.629840 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.629855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.631100 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.631106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.632274 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.632288 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.632941 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.632945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.633068 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.633703 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:13.633721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:13.636486 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":267,"threads_started":1}
W20260809 05:22:13.636907 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:13.628390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:53573 (local address 127.22.1.126:41931)
0809 05:22:13.628574 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:13.628579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:13.628847 (+ 268us) server_negotiation.cc:407] Connection header received
0809 05:22:13.629003 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:13.629007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:13.629086 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:13.629202 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:13.629982 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.630956 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.632400 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.632830 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.632929 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.636677 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:53573: BlockingRecv error: recv got EOF from 127.22.1.67:53573 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:13 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:13.698523 4061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:13.690525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:40313)
0809 05:22:13.691042 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:13.691064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:13.691208 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:13.691552 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:13.691555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:13.691983 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:13.692219 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.692230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.693286 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.693290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.694204 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:13.694215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.694865 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.694870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:13.695003 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.695596 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:13.695614 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:13.698264 (+ 2650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":263,"threads_started":1}
W20260809 05:22:13.698697 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:13.690602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:40313 (local address 127.22.1.126:41931)
0809 05:22:13.690822 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:13.690828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:13.691142 (+ 314us) server_negotiation.cc:407] Connection header received
0809 05:22:13.691321 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:13.691325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:13.691386 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:13.691481 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:13.692384 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.693145 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.694327 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:13.694738 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:13.694831 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:13.698473 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:40313: BlockingRecv error: recv got EOF from 127.22.1.65:40313 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:14.154899 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:14.146784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:54257)
0809 05:22:14.147290 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:14.147315 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:14.147447 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:14.147777 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:14.147781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:14.148218 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:14.148413 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.148424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.149495 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.149499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.150414 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.150426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.151141 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.151145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.151278 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.151880 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:14.151898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:14.154646 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
W20260809 05:22:14.155053 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:14.146881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:54257 (local address 127.22.1.126:41931)
0809 05:22:14.147095 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:14.147101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:14.147406 (+ 305us) server_negotiation.cc:407] Connection header received
0809 05:22:14.147578 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:14.147584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:14.147658 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:14.147741 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:14.148557 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.149393 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.150582 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.151018 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.151153 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.154843 (+ 3690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:54257: BlockingRecv error: recv got EOF from 127.22.1.66:54257 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:14.646296 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:14.637964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:50907)
0809 05:22:14.638493 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:14.638511 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:14.638636 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:14.638937 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:14.638940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:14.639291 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:14.639494 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.639505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.640583 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.640587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.641691 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.641706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.642416 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.642422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.642613 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.643181 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:14.643199 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:14.646037 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
W20260809 05:22:14.646467 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:14.638094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:50907 (local address 127.22.1.126:41931)
0809 05:22:14.638263 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:14.638268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:14.638582 (+ 314us) server_negotiation.cc:407] Connection header received
0809 05:22:14.638746 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:14.638749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:14.638815 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:14.638907 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:14.639680 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.640440 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.641833 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.642267 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.642381 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.646266 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:50907: BlockingRecv error: recv got EOF from 127.22.1.67:50907 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:14 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:14.708432 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:14.699576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:41079)
0809 05:22:14.700110 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:14.700128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:14.700260 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:14.700611 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:14.700615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:14.701012 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:14.701197 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.701207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.702289 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.702293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.703409 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:14.703421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.704312 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.704317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:14.704509 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.705230 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:14.705252 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:14.708193 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":302,"threads_started":1}
W20260809 05:22:14.708675 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:14.699740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:41079 (local address 127.22.1.126:41931)
0809 05:22:14.699995 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:14.700001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:14.700219 (+ 218us) server_negotiation.cc:407] Connection header received
0809 05:22:14.700358 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:14.700362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:14.700428 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:14.700546 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:14.701339 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.702176 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.703537 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:14.704144 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:14.704279 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:14.708425 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:41079: BlockingRecv error: recv got EOF from 127.22.1.65:41079 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:15.164548 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:15.156064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:53045)
0809 05:22:15.156571 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:15.156596 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:15.156721 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:15.157068 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:15.157072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:15.157669 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:15.157832 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.157841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.158806 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.158811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.159803 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.159816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.160536 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.160541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.160677 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.161337 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:15.161357 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:15.164259 (+ 2902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":260,"threads_started":1}
W20260809 05:22:15.164729 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:15.156152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:53045 (local address 127.22.1.126:41931)
0809 05:22:15.156357 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:15.156363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:15.156692 (+ 329us) server_negotiation.cc:407] Connection header received
0809 05:22:15.156831 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:15.156836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:15.156919 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:15.157000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:15.157960 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.158693 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.159950 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.160379 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.160473 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.164505 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:53045: BlockingRecv error: recv got EOF from 127.22.1.66:53045 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:15.655830 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:15.647531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:47987)
0809 05:22:15.648093 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:15.648114 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:15.648255 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:15.648682 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:15.648686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:15.649229 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:15.649399 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.649408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.650412 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.650415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.651397 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.651408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.652101 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.652105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.652238 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.652836 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:15.652855 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:15.655590 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":295,"threads_started":1}
W20260809 05:22:15.656095 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:15.647628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47987 (local address 127.22.1.126:41931)
0809 05:22:15.647838 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:15.647845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:15.648209 (+ 364us) server_negotiation.cc:407] Connection header received
0809 05:22:15.648388 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:15.648393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:15.648522 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:15.648628 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:15.649560 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.650307 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.651539 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.651982 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.652078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.655822 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:47987: BlockingRecv error: recv got EOF from 127.22.1.67:47987 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:15 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:15.717078 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:15.709394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:55995)
0809 05:22:15.709906 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:15.709922 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:15.710035 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:15.710321 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:15.710325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:15.710748 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:15.710915 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.710923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.711891 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.711895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.712828 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:15.712839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.713585 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.713589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:15.713713 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.714251 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:15.714269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:15.716836 (+ 2567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":246,"threads_started":1}
W20260809 05:22:15.717252 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:15.709459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:55995 (local address 127.22.1.126:41931)
0809 05:22:15.709650 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:15.709655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:15.709985 (+ 330us) server_negotiation.cc:407] Connection header received
0809 05:22:15.710142 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:15.710147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:15.710208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:15.710286 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:15.711035 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.711767 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.712952 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:15.713455 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:15.713556 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:15.717033 (+ 3477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:55995: BlockingRecv error: recv got EOF from 127.22.1.65:55995 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:16.174265 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:16.165707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:54397)
0809 05:22:16.166264 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:16.166286 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:16.166408 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:16.166703 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:16.166706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:16.167110 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:16.167273 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.167282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.168532 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.168538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.169729 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.169743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.170548 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.170554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.170687 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.171274 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:16.171292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:16.174013 (+ 2721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":281,"threads_started":1}
W20260809 05:22:16.174490 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:16.165972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:54397 (local address 127.22.1.126:41931)
0809 05:22:16.166203 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:16.166210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:16.166359 (+ 149us) server_negotiation.cc:407] Connection header received
0809 05:22:16.166505 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:16.166509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:16.166576 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:16.166663 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:16.167440 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.168374 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.169845 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.170364 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.170463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.174252 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:54397: BlockingRecv error: recv got EOF from 127.22.1.66:54397 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:16.665329 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:16.656855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:43057)
0809 05:22:16.657436 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:16.657453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:16.657590 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:16.657931 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:16.657936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:16.658392 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:16.658565 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.658576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.659586 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.659591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.660518 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.660531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.661275 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.661281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.661440 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.661993 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:16.662021 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:16.665038 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":284,"threads_started":1}
W20260809 05:22:16.665552 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:16.657041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:43057 (local address 127.22.1.126:41931)
0809 05:22:16.657314 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:16.657321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:16.657567 (+ 246us) server_negotiation.cc:407] Connection header received
0809 05:22:16.657719 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:16.657724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:16.657790 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:16.657873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:16.658720 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.659437 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.660661 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.661094 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.661240 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.665267 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:43057: BlockingRecv error: recv got EOF from 127.22.1.67:43057 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:16 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:16.726691 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:16.718120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:53351)
0809 05:22:16.718608 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:16.718626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:16.718829 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:16.719148 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:16.719153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:16.719557 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:16.719742 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.719752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.720765 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.720768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.721703 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:16.721714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.722517 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.722523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:16.722648 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.723213 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:16.723231 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:16.726438 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":259,"threads_started":1}
W20260809 05:22:16.726850 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:16.718261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:53351 (local address 127.22.1.126:41931)
0809 05:22:16.718489 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:16.718496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:16.718726 (+ 230us) server_negotiation.cc:407] Connection header received
0809 05:22:16.718941 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:16.718946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:16.719029 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:16.719111 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:16.719883 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.720653 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.721908 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:16.722380 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:16.722497 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:16.726629 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:53351: BlockingRecv error: recv got EOF from 127.22.1.65:53351 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:17.183611 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:17.175273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:58587)
0809 05:22:17.175789 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:17.175811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:17.175933 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:17.176279 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:17.176282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:17.176659 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:17.176823 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.176832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.178045 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.178052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.179138 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.179152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.179920 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.179925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.180051 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.180656 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:17.180673 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:17.183370 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":288,"threads_started":1}
W20260809 05:22:17.183924 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:17.175462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:58587 (local address 127.22.1.126:41931)
0809 05:22:17.175639 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:17.175644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:17.175879 (+ 235us) server_negotiation.cc:407] Connection header received
0809 05:22:17.176073 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:17.176078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:17.176145 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:17.176251 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:17.176982 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.177882 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.179303 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.179780 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.179889 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.183658 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:58587: BlockingRecv error: recv got EOF from 127.22.1.66:58587 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:17.678282 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:17.666320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:53535)
0809 05:22:17.667204 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:17.667221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:17.667354 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:17.667778 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:17.667783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:17.668318 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:17.668542 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.668555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.669588 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.669592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.670550 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.670563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.671327 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.671332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.671457 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.675013 (+ 3556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:17.675042 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:17.678056 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":595,"threads_started":1}
W20260809 05:22:17.678545 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:17.666454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:53535 (local address 127.22.1.126:41931)
0809 05:22:17.666702 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:17.666709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:17.667307 (+ 598us) server_negotiation.cc:407] Connection header received
0809 05:22:17.667474 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:17.667480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:17.667619 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:17.667724 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:17.668672 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.669421 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.670673 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.671170 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.671265 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.678286 (+ 7021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:53535: BlockingRecv error: recv got EOF from 127.22.1.67:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:17 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:17.736433 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:17.727744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:54849)
0809 05:22:17.728579 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:17.728600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:17.728745 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:17.729083 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:17.729088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:17.729593 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:17.729774 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.729783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.730836 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.730842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.731771 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:17.731783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.732475 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.732481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:17.732658 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.733273 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:17.733291 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:17.736163 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":411,"threads_started":1}
W20260809 05:22:17.736652 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:17.727896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:54849 (local address 127.22.1.126:41931)
0809 05:22:17.728098 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:17.728104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:17.728690 (+ 586us) server_negotiation.cc:407] Connection header received
0809 05:22:17.728865 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:17.728871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:17.728944 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:17.729034 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:17.729926 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.730690 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.731911 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:17.732328 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:17.732435 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:17.736364 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:54849: BlockingRecv error: recv got EOF from 127.22.1.65:54849 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:18.192659 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:18.184536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:49373)
0809 05:22:18.185137 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:18.185184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:18.185297 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:18.185690 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:18.185693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:18.186103 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:18.186274 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.186283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.187275 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.187280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.188211 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.188223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.188859 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.188863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.188974 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.189584 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:18.189601 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:18.192370 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":257,"threads_started":1}
W20260809 05:22:18.192790 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:18.184668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49373 (local address 127.22.1.126:41931)
0809 05:22:18.184916 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:18.184921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:18.185269 (+ 348us) server_negotiation.cc:407] Connection header received
0809 05:22:18.185431 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:18.185435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:18.185541 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:18.185623 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:18.186387 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.187134 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.188343 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.188743 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.188834 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.192580 (+ 3746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:49373: BlockingRecv error: recv got EOF from 127.22.1.66:49373 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:18.688453 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:18.679481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:50653)
0809 05:22:18.680045 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:18.680061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:18.680181 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:18.680536 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:18.680540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:18.680947 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:18.681129 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.681138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.682450 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.682455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.683653 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.683667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.684387 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.684392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.684542 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.685177 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:18.685197 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:18.688200 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":249,"threads_started":1}
W20260809 05:22:18.688661 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:18.679626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:50653 (local address 127.22.1.126:41931)
0809 05:22:18.679818 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:18.679822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:18.680148 (+ 326us) server_negotiation.cc:407] Connection header received
0809 05:22:18.680280 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:18.680284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:18.680369 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:18.680497 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:18.681301 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.682308 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.683794 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.684254 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.684352 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.688428 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:50653: BlockingRecv error: recv got EOF from 127.22.1.67:50653 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:18 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:18.746215 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:18.737574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:34961)
0809 05:22:18.738155 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:18.738172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:18.738309 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:18.738630 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:18.738636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:18.739053 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:18.739214 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.739222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.740272 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.740276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.741265 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:18.741279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.742045 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.742052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:18.742217 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.742819 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:18.742842 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:18.745957 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":304,"threads_started":1}
W20260809 05:22:18.746404 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:18.737764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:34961 (local address 127.22.1.126:41931)
0809 05:22:18.737974 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:18.737981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:18.738261 (+ 280us) server_negotiation.cc:407] Connection header received
0809 05:22:18.738426 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:18.738430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:18.738497 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:18.738580 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:18.739364 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.740121 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.741403 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:18.741863 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:18.741980 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:18.746165 (+ 4185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:34961: BlockingRecv error: recv got EOF from 127.22.1.65:34961 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:19.202143 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:19.193621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:57025)
0809 05:22:19.194138 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:19.194157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:19.194291 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:19.194640 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:19.194645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:19.195087 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:19.195240 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.195249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.196504 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.196509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.197666 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.197680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.198383 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.198387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.198522 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.199061 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:19.199077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:19.201845 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
W20260809 05:22:19.202277 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:19.193784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57025 (local address 127.22.1.126:41931)
0809 05:22:19.194009 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:19.194015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:19.194238 (+ 223us) server_negotiation.cc:407] Connection header received
0809 05:22:19.194411 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:19.194415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:19.194477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:19.194588 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:19.195385 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.196356 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.197800 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.198231 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.198328 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.202056 (+ 3728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:57025: BlockingRecv error: recv got EOF from 127.22.1.66:57025 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:19.697562 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:19.689481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:50349)
0809 05:22:19.689991 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:19.690009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:19.690131 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:19.690414 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:19.690417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:19.690829 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:19.690987 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.690996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.691974 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.691978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.692923 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.692937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.693751 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.693756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.693884 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.694456 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:19.694472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:19.697317 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
W20260809 05:22:19.697815 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:19.689605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:50349 (local address 127.22.1.126:41931)
0809 05:22:19.689846 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:19.689853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:19.690083 (+ 230us) server_negotiation.cc:407] Connection header received
0809 05:22:19.690217 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:19.690220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:19.690294 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:19.690373 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:19.691101 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.691831 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.693062 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.693579 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.693674 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.697536 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:50349: BlockingRecv error: recv got EOF from 127.22.1.67:50349 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:19 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:19.755975 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:19.747269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:56583)
0809 05:22:19.747947 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:19.747964 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:19.748113 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:19.748553 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:19.748560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:19.748998 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:19.749237 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.749252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.750280 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.750284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.751273 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:19.751285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.752021 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.752025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:19.752152 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.752719 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:19.752737 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:19.755709 (+ 2972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":386,"threads_started":1}
W20260809 05:22:19.756115 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:19.747294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:56583 (local address 127.22.1.126:41931)
0809 05:22:19.747518 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:19.747525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:19.748056 (+ 531us) server_negotiation.cc:407] Connection header received
0809 05:22:19.748238 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:19.748243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:19.748323 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:19.748427 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:19.749398 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.750150 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.751415 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:19.751871 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:19.751983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:19.755915 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:56583: BlockingRecv error: recv got EOF from 127.22.1.65:56583 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:20.212028 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:20.203286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:57847)
0809 05:22:20.203934 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:20.203955 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:20.204093 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:20.204367 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:20.204370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:20.204782 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:20.204955 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.204965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.206189 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.206193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.207224 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.207240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.207915 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.207919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.208055 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.208735 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:20.208755 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:20.211769 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":384,"threads_started":1}
W20260809 05:22:20.212240 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:20.203379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57847 (local address 127.22.1.126:41931)
0809 05:22:20.203571 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:20.203578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:20.204062 (+ 484us) server_negotiation.cc:407] Connection header received
0809 05:22:20.204187 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:20.204190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:20.204256 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:20.204335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:20.205142 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.206068 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.207354 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.207791 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.207889 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.211993 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:57847: BlockingRecv error: recv got EOF from 127.22.1.66:57847 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:20.707237 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:20.698434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:55207)
0809 05:22:20.699003 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:20.699019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:20.699157 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:20.699529 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:20.699534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:20.699953 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:20.700121 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.700132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.701257 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.701261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.702223 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.702237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.702948 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.702954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.703123 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.703721 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:20.703741 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:20.706897 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":310,"threads_started":1}
W20260809 05:22:20.707367 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:20.698532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:55207 (local address 127.22.1.126:41931)
0809 05:22:20.698747 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:20.698754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:20.699106 (+ 352us) server_negotiation.cc:407] Connection header received
0809 05:22:20.699273 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:20.699279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:20.699358 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:20.699444 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:20.700279 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.701069 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.702367 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.702787 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.702902 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.707139 (+ 4237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:55207: BlockingRecv error: recv got EOF from 127.22.1.67:55207 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:20 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:20.765460 4097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:20.757248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:59617)
0809 05:22:20.757789 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:20.757810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:20.757949 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:20.758282 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:20.758286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:20.758692 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:20.758842 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.758850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.759905 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.759911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.760841 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:20.760853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.761637 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.761643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:20.761812 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.762401 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:20.762419 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:20.765209 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":280,"threads_started":1}
W20260809 05:22:20.765650 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:20.757326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:59617 (local address 127.22.1.126:41931)
0809 05:22:20.757564 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:20.757571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:20.757896 (+ 325us) server_negotiation.cc:407] Connection header received
0809 05:22:20.758060 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:20.758064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:20.758130 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:20.758223 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:20.758996 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.759766 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.760985 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:20.761476 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:20.761600 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:20.765413 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:59617: BlockingRecv error: recv got EOF from 127.22.1.65:59617 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:21.222064 4100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:21.213121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:54813)
0809 05:22:21.213732 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:21.213755 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:21.213881 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:21.214339 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:21.214343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:21.214771 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:21.214928 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.214939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.216147 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.216153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.217176 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.217191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.218139 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.218144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.218269 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.218831 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:21.218848 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:21.221822 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":336,"threads_started":1}
W20260809 05:22:21.222265 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:21.213314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:54813 (local address 127.22.1.126:41931)
0809 05:22:21.213497 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:21.213502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:21.213849 (+ 347us) server_negotiation.cc:407] Connection header received
0809 05:22:21.214082 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:21.214090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:21.214172 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:21.214281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:21.215071 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.216002 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.217337 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.217989 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.218106 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.222046 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:54813: BlockingRecv error: recv got EOF from 127.22.1.66:54813 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:21.716674 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:21.708305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:34739)
0809 05:22:21.708812 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:21.708833 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:21.708975 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:21.709375 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:21.709380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:21.709808 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:21.709961 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.709970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.711066 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.711071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.712050 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.712064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.712794 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.712800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.712972 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.713644 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:21.713662 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:21.716416 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":277,"threads_started":1}
W20260809 05:22:21.716915 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:21.708567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:34739 (local address 127.22.1.126:41931)
0809 05:22:21.708798 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:21.708805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:21.708915 (+ 110us) server_negotiation.cc:407] Connection header received
0809 05:22:21.709107 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:21.709112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:21.709237 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:21.709339 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:21.710123 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.710913 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.712201 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.712640 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.712753 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.716668 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:34739: BlockingRecv error: recv got EOF from 127.22.1.67:34739 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:21 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:21.774799 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:21.766376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:36315)
0809 05:22:21.766832 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:21.766849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:21.766966 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:21.767239 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:21.767242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:21.767653 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:21.767830 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.767838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.768822 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.768826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.769806 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:21.769819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.770599 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.770604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:21.770726 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.771332 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:21.771351 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:21.774507 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
W20260809 05:22:21.774986 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:21.766480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:36315 (local address 127.22.1.126:41931)
0809 05:22:21.766720 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:21.766726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:21.766923 (+ 197us) server_negotiation.cc:407] Connection header received
0809 05:22:21.767062 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:21.767065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:21.767129 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:21.767210 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:21.767962 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.768708 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.769943 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:21.770441 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:21.770546 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:21.774769 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:36315: BlockingRecv error: recv got EOF from 127.22.1.65:36315 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:22.231366 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:22.222956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:43483)
0809 05:22:22.223464 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:22.223488 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:22.223608 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:22.223885 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:22.223888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:22.224301 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:22.224513 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.224524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.225574 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.225579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.226577 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.226591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.227306 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.227311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.227445 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.228066 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:22.228085 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:22.231072 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
W20260809 05:22:22.231521 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:22.223093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:43483 (local address 127.22.1.126:41931)
0809 05:22:22.223276 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:22.223281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:22.223555 (+ 274us) server_negotiation.cc:407] Connection header received
0809 05:22:22.223703 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:22.223707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:22.223771 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:22.223858 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:22.224639 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.225440 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.226724 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.227168 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.227293 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.231279 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:43483: BlockingRecv error: recv got EOF from 127.22.1.66:43483 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:22.728698 4105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:22.717776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:50417)
0809 05:22:22.718289 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:22.718310 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:22.718453 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:22.718878 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:22.718883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:22.719427 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:22.719648 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.719658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.720638 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.720641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.721598 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.721610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.722246 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.722251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.722366 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.725587 (+ 3221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:22.725606 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:22.728394 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":328,"threads_started":1}
W20260809 05:22:22.728876 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:22.717865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:50417 (local address 127.22.1.126:41931)
0809 05:22:22.718072 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:22.718077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:22.718380 (+ 303us) server_negotiation.cc:407] Connection header received
0809 05:22:22.718624 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:22.718629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:22.718734 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:22.718834 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:22.719781 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.720536 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.721716 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.722116 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.722209 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.728617 (+ 6408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:50417: BlockingRecv error: recv got EOF from 127.22.1.67:50417 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:22 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:22.784025 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:22.775685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:36217)
0809 05:22:22.776231 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:22.776247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:22.776369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:22.776688 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:22.776691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:22.777102 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:22.777297 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.777306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.778325 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.778329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.779624 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:22.779638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.780397 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.780402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:22.780534 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.781096 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:22.781115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:22.783788 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":296,"threads_started":1}
W20260809 05:22:22.784204 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:22.775765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:36217 (local address 127.22.1.126:41931)
0809 05:22:22.775981 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:22.775986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:22.776354 (+ 368us) server_negotiation.cc:407] Connection header received
0809 05:22:22.776473 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:22.776478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:22.776547 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:22.776636 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:22.777425 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.778159 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.779733 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:22.780260 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:22.780354 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:22.783997 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:36217: BlockingRecv error: recv got EOF from 127.22.1.65:36217 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:23.240373 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:23.232343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:40889)
0809 05:22:23.232830 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:23.232851 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:23.232987 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:23.233358 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:23.233362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:23.233781 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:23.233989 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.234001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.235043 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.235047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.235958 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.235969 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.236669 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.236673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.236796 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.237387 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:23.237406 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:23.240133 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":256,"threads_started":1}
W20260809 05:22:23.240554 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:23.232413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:40889 (local address 127.22.1.126:41931)
0809 05:22:23.232614 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:23.232621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:23.232936 (+ 315us) server_negotiation.cc:407] Connection header received
0809 05:22:23.233103 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:23.233108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:23.233205 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:23.233288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:23.234163 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.234922 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.236098 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.236556 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.236652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.240357 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:40889: BlockingRecv error: recv got EOF from 127.22.1.66:40889 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:23.738178 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:23.729754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:56189)
0809 05:22:23.730298 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:23.730313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:23.730478 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:23.730818 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:23.730823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:23.731256 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:23.731413 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.731422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.732466 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.732472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.733421 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.733443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.734149 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.734155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.734330 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.734882 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:23.734899 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:23.737855 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":293,"threads_started":1}
W20260809 05:22:23.738337 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:23.729868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:56189 (local address 127.22.1.126:41931)
0809 05:22:23.730104 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:23.730110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:23.730395 (+ 285us) server_negotiation.cc:407] Connection header received
0809 05:22:23.730599 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:23.730603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:23.730678 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:23.730758 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:23.731566 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.732306 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.733567 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.734010 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.734122 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.738085 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:56189: BlockingRecv error: recv got EOF from 127.22.1.67:56189 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:23 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:23.793509 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:23.785053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:42313)
0809 05:22:23.785799 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:23.785816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:23.785949 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:23.786277 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:23.786281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:23.786701 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:23.786864 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.786874 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.787878 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.787882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.788811 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:23.788823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.789636 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.789642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:23.789793 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.790376 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:23.790395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:23.793170 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":478,"threads_started":1}
W20260809 05:22:23.793617 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:23.785137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:42313 (local address 127.22.1.126:41931)
0809 05:22:23.785376 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:23.785384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:23.785899 (+ 515us) server_negotiation.cc:407] Connection header received
0809 05:22:23.786070 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:23.786076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:23.786139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:23.786217 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:23.787005 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.787739 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.788949 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:23.789443 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:23.789565 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:23.793391 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:42313: BlockingRecv error: recv got EOF from 127.22.1.65:42313 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:24.249745 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:24.241323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:59679)
0809 05:22:24.241897 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:24.241917 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:24.242045 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:24.242320 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:24.242323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:24.242714 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:24.242872 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.242881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.243943 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.243947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.244914 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.244927 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.245751 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.245757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.245889 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.246490 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:24.246510 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:24.249411 (+ 2901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":318,"threads_started":1}
W20260809 05:22:24.249902 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:24.241446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:59679 (local address 127.22.1.126:41931)
0809 05:22:24.241652 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:24.241657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:24.241986 (+ 329us) server_negotiation.cc:407] Connection header received
0809 05:22:24.242138 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:24.242142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:24.242203 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:24.242293 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:24.243030 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.243815 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.245072 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.245547 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.245650 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.249646 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:59679: BlockingRecv error: recv got EOF from 127.22.1.66:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:24.748039 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:24.739173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:45189)
0809 05:22:24.739776 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:24.739792 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:24.739909 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:24.740265 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:24.740269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:24.740694 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:24.740856 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.740866 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.741968 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.741973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.742928 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.742941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.743727 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.743731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.743854 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.744427 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:24.744486 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:24.747726 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":331,"threads_started":1}
W20260809 05:22:24.748157 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:24.739289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:45189 (local address 127.22.1.126:41931)
0809 05:22:24.739482 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:24.739487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:24.739873 (+ 386us) server_negotiation.cc:407] Connection header received
0809 05:22:24.740010 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:24.740013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:24.740092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:24.740183 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:24.741026 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.741812 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.743107 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.743545 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.743654 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.747955 (+ 4301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:45189: BlockingRecv error: recv got EOF from 127.22.1.67:45189 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:24 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:24.803334 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:24.794668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:60101)
0809 05:22:24.795194 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:24.795216 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:24.795373 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:24.795771 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:24.795776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:24.796210 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:24.796441 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.796451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.797675 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.797679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.798620 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:24.798631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.799329 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.799335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:24.799500 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.800117 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:24.800136 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:24.802986 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":244,"threads_started":1}
W20260809 05:22:24.803500 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:24.794844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:60101 (local address 127.22.1.126:41931)
0809 05:22:24.795193 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:24.795198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:24.795302 (+ 104us) server_negotiation.cc:407] Connection header received
0809 05:22:24.795532 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:24.795538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:24.795626 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:24.795727 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:24.796596 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.797528 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.798761 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:24.799185 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:24.799294 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:24.803257 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:60101: BlockingRecv error: recv got EOF from 127.22.1.65:60101 (error 108)
Metrics: {"server-negotiator.queue_time_us":203}
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:25.259104 4116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:25.250823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:54251)
0809 05:22:25.251327 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:25.251347 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:25.251482 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:25.251797 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:25.251802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:25.252282 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:25.252505 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.252518 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.253691 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.253697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.254619 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.254631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.255307 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.255312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.255474 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.256077 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:25.256095 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:25.258749 (+ 2654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
W20260809 05:22:25.259193 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:25.250991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:54251 (local address 127.22.1.126:41931)
0809 05:22:25.251195 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:25.251202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:25.251428 (+ 226us) server_negotiation.cc:407] Connection header received
0809 05:22:25.251603 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:25.251607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:25.251669 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:25.251751 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:25.252662 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.253556 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.254751 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.255165 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.255279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.258986 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:54251: BlockingRecv error: recv got EOF from 127.22.1.66:54251 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.67@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:25.757709 4117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:25.749283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.67:40081)
0809 05:22:25.749754 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:25.749771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:25.749929 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:25.750337 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:25.750343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:25.750806 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:25.751072 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.751083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.752168 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.752174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.753106 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.753119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.753852 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.753856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.753986 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.754581 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:25.754598 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:25.757439 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
W20260809 05:22:25.757970 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:25.749425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:40081 (local address 127.22.1.126:41931)
0809 05:22:25.749663 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:25.749669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:25.749852 (+ 183us) server_negotiation.cc:407] Connection header received
0809 05:22:25.750099 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:25.750103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:25.750177 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:25.750271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:25.751213 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.752016 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.753277 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.753727 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.753828 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.757715 (+ 3887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.67:40081: BlockingRecv error: recv got EOF from 127.22.1.67:40081 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:25 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.65@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:25.813119 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:25.804500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.65:60759)
0809 05:22:25.804992 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:25.805010 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:25.805138 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:25.805466 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:25.805469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:25.805933 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:25.806168 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.806180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.807179 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.807185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.808142 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:25.808156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.808934 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.808939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:25.809112 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.809767 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:25.809785 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:25.812772 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
W20260809 05:22:25.813247 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:25.804606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:60759 (local address 127.22.1.126:41931)
0809 05:22:25.804825 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:25.804831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:25.805096 (+ 265us) server_negotiation.cc:407] Connection header received
0809 05:22:25.805276 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:25.805280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:25.805343 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:25.805426 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:25.806310 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.807044 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.808299 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:25.808766 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:25.808872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:25.813029 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.65:60759: BlockingRecv error: recv got EOF from 127.22.1.65:60759 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 09 05:22:26 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
Aug 09 05:22:26 dist-test-slave-rmhg krb5kdc[2482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.1.66@KRBTEST.COM for oryx/127.22.1.126@KRBTEST.COM, Server not found in Kerberos database
I20260809 05:22:26.268488 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:26.260260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41931 (local address 127.22.1.66:49111)
0809 05:22:26.260786 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:26.260812 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:26.260926 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:26.261305 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:26.261309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:26.261759 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:26.261959 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:26.261970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.263009 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.263013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:26.264029 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:26.264044 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.264785 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.264790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:26.264908 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:26.265494 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:26.265511 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:26.268223 (+ 2712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.1.126:41931: Server oryx/127.22.1.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":311,"threads_started":1}
W20260809 05:22:26.268659 3653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:26.260394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49111 (local address 127.22.1.126:41931)
0809 05:22:26.260582 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:26.260587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:26.260910 (+ 323us) server_negotiation.cc:407] Connection header received
0809 05:22:26.261025 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:26.261029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:26.261104 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:26.261234 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:26.262111 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.262874 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.264181 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.264653 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.264747 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:26.268419 (+ 3672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.1.66:49111: BlockingRecv error: recv got EOF from 127.22.1.66:49111 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:22:26.587248 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3400
I20260809 05:22:26.599901 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:26.588424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.22.1.65:38353)
0809 05:22:26.588976 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:26.588985 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:26.589018 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:22:26.589083 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:26.589089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:26.589370 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:26.589558 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:26.591057 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.592091 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.593489 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.594087 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.594197 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:26.596750 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:26.596760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:26.596775 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:26.596810 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:26.599337 (+ 2527us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 05:22:26.599666 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37160: Request ticket server kudu/127.22.1.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":228,"threads_started":1}
W20260809 05:22:26.600013 4122 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37160: Request ticket server kudu/127.22.1.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 05:22:26.600214 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3400: Not authorized: Client connection negotiation failed: client connection to 127.22.1.65:38353: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 05:22:26.600411 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 3400
I20260809 05:22:26.633323 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3655
I20260809 05:22:26.646260 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:26.634418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.22.1.66:45201)
0809 05:22:26.635039 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:26.635047 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:26.635081 (+ 34us) server_negotiation.cc:407] Connection header received
0809 05:22:26.635147 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:26.635154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:26.635385 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:26.635581 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:26.636947 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.638100 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.639398 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.639975 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.640099 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:26.642758 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:26.642769 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:26.642786 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:26.642827 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:26.645551 (+ 2724us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 05:22:26.645914 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51942: Request ticket server kudu/127.22.1.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
W20260809 05:22:26.646390 4123 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51942: Request ticket server kudu/127.22.1.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 05:22:26.646410 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3655: Not authorized: Client connection negotiation failed: client connection to 127.22.1.66:45201: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 05:22:26.646649 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 3655
I20260809 05:22:26.683200 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3855
I20260809 05:22:26.695820 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:26.684284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54908 (local address 127.22.1.67:37079)
0809 05:22:26.684912 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:26.684919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:26.684946 (+ 27us) server_negotiation.cc:407] Connection header received
0809 05:22:26.685009 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:26.685015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:26.685305 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:26.685489 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:26.686926 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.688034 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.689322 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:26.689914 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:26.690036 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:26.692574 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:26.692584 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:26.692599 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:26.692651 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:26.695148 (+ 2497us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0809 05:22:26.695481 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54908: Request ticket server kudu/127.22.1.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":357,"threads_started":1}
W20260809 05:22:26.695942 4124 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54908: Request ticket server kudu/127.22.1.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260809 05:22:26.695961 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3855: Not authorized: Client connection negotiation failed: client connection to 127.22.1.67:37079: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.22.1.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260809 05:22:26.696128 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 3855
I20260809 05:22:26.720388 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 3301
I20260809 05:22:26.721825 3390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:26.839884 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 3301
2026-08-09T05:22:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21417 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-09T05:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:26Z Disabled control of system clock
I20260809 05:22:26.931797 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38343
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260809 05:22:27.281291 4132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:27.281616 4132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:27.281705 4132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:27.289085 4132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:27.289237 4132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:27.289276 4132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:27.289333 4132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:27.289377 4132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:27.298900 4132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:27.300297 4132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:27.302233 4132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:27.314561 4143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:27.314595 4145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:27.314618 4142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:27.315897 4132 server_base.cc:1034] running on GCE node
I20260809 05:22:27.316816 4132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:27.318451 4132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:27.319690 4132 hybrid_clock.cc:648] HybridClock initialized: now 1786252947319643 us; error 32 us; skew 500 ppm
I20260809 05:22:27.320086 4132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:27.322531 4132 webserver.cc:460] Webserver started at http://127.22.1.126:33323/ using document root <none> and password file <none>
I20260809 05:22:27.323297 4132 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:27.323410 4132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:27.323776 4132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:27.327083 4132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "4422adcc55c9451e8707d9963d425600"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "27b7506bf83786560b57f147ee5c710d"
server_key_iv: "386cc1fd47f1de87ea71268a547d3742"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.327962 4132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "4422adcc55c9451e8707d9963d425600"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "27b7506bf83786560b57f147ee5c710d"
server_key_iv: "386cc1fd47f1de87ea71268a547d3742"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.333736 4132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260809 05:22:27.337718 4158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:27.339203 4132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:22:27.339399 4132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "4422adcc55c9451e8707d9963d425600"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "27b7506bf83786560b57f147ee5c710d"
server_key_iv: "386cc1fd47f1de87ea71268a547d3742"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.339675 4132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:27.388618 4132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:27.389741 4132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:27.390085 4132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:27.408835 4132 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38343
I20260809 05:22:27.408872 4237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38343 every 8 connection(s)
I20260809 05:22:27.410393 4132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:22:27.414412 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:27.418867 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 4132
I20260809 05:22:27.419121 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:22:27.419994 22533 external_mini_cluster.cc:1428] Setting key 0d9d7a41d21dac7c217ddb6dc4765b27
I20260809 05:22:27.427065 4238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600: Bootstrap starting.
I20260809 05:22:27.431020 4238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:27.432339 4238 log.cc:824] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:27.435958 4238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600: No bootstrap required, opened a new log
I20260809 05:22:27.436065 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:27.422409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.22.1.126:38343)
0809 05:22:27.423616 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:27.423630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:27.423678 (+ 48us) server_negotiation.cc:407] Connection header received
0809 05:22:27.425644 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:27.425675 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:27.426268 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:27.426658 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:27.427523 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:27.428916 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:27.430845 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:27.431696 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:27.431860 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:27.433266 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:27.433283 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:27.433306 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:27.434110 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:27.434200 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:27.434223 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:27.434322 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:27.434573 (+ 251us) server_negotiation.cc:299] Negotiation successful
0809 05:22:27.434931 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":380,"threads_started":1}
I20260809 05:22:27.443655 4238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } }
I20260809 05:22:27.444116 4238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:27.444233 4238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4422adcc55c9451e8707d9963d425600, State: Initialized, Role: FOLLOWER
I20260809 05:22:27.444885 4238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } }
I20260809 05:22:27.445174 4238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:22:27.445334 4238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:22:27.445530 4238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:27.447835 4238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } }
I20260809 05:22:27.448386 4238 leader_election.cc:302] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4422adcc55c9451e8707d9963d425600; no voters:
I20260809 05:22:27.448995 4238 leader_election.cc:288] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:22:27.449143 4247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:27.449855 4247 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [term 1 LEADER]: Becoming Leader. State: Replica: 4422adcc55c9451e8707d9963d425600, State: Running, Role: LEADER
I20260809 05:22:27.450449 4247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } }
I20260809 05:22:27.451134 4238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:22:27.453843 4249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4422adcc55c9451e8707d9963d425600. Latest consensus state: current_term: 1 leader_uuid: "4422adcc55c9451e8707d9963d425600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } } }
I20260809 05:22:27.454255 4249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:27.453750 4248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4422adcc55c9451e8707d9963d425600" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4422adcc55c9451e8707d9963d425600" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38343 } } }
I20260809 05:22:27.454800 4248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600 [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:27.460278 4255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:22:27.466082 4255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:22:27.476089 4255 catalog_manager.cc:1347] Generated new cluster ID: f0b9bf69de4d444ca4d0af9a8588a5d9
I20260809 05:22:27.476227 4255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:22:27.495414 4255 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:22:27.496814 4255 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:22:27.509300 4255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4422adcc55c9451e8707d9963d425600: Generated new TSK 0
I20260809 05:22:27.510412 4255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:22:27.528695 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--builtin_ntp_servers=127.22.1.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260809 05:22:27.901613 4277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:27.901919 4277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:27.902004 4277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:27.909327 4277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:27.909442 4277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:27.909591 4277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:27.918982 4277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:27.920269 4277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:27.922278 4277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:27.936272 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:27.936303 4287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:27.936383 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:27.937443 4277 server_base.cc:1034] running on GCE node
I20260809 05:22:27.938122 4277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:27.939167 4277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:27.940398 4277 hybrid_clock.cc:648] HybridClock initialized: now 1786252947940332 us; error 57 us; skew 500 ppm
I20260809 05:22:27.940769 4277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:27.949661 4277 webserver.cc:460] Webserver started at http://127.22.1.65:38785/ using document root <none> and password file <none>
I20260809 05:22:27.950474 4277 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:27.950596 4277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:27.950969 4277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:27.954211 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "76fd95205b3f44c48032b19a9352dea7"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "08f1039a911fdebc33e16f8eeb21ac15"
server_key_iv: "48c64e8964dd99846676d70f83a1f190"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.954995 4277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "76fd95205b3f44c48032b19a9352dea7"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "08f1039a911fdebc33e16f8eeb21ac15"
server_key_iv: "48c64e8964dd99846676d70f83a1f190"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.960570 4277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260809 05:22:27.964244 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:27.965705 4277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:27.965869 4277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "76fd95205b3f44c48032b19a9352dea7"
format_stamp: "Formatted at 2026-08-09 05:22:27 on dist-test-slave-rmhg"
server_key: "08f1039a911fdebc33e16f8eeb21ac15"
server_key_iv: "48c64e8964dd99846676d70f83a1f190"
server_key_version: "encryptionkey@0"
I20260809 05:22:27.966074 4277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:27.990737 4277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:27.991641 4277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:27.991912 4277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:27.992919 4277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:27.994479 4277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:27.994575 4277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:27.994683 4277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:27.994752 4277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.025928 4277 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:45227
I20260809 05:22:28.026016 4481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:45227 every 8 connection(s)
I20260809 05:22:28.027288 4277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:28.031499 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 4277
I20260809 05:22:28.031739 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:22:28.032590 22533 external_mini_cluster.cc:1428] Setting key 22db29b0bb35f49619cb45a4c10b863f
I20260809 05:22:28.035809 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--builtin_ntp_servers=127.22.1.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 05:22:28.042513 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.030119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:45359 (local address 127.22.1.126:38343)
0809 05:22:28.030622 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:28.030629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:28.032161 (+ 1532us) server_negotiation.cc:407] Connection header received
0809 05:22:28.034811 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:28.034815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:28.034881 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:28.034983 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:28.036530 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.037299 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.039504 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.039916 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.040046 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.041692 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:28.041698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:28.041701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:28.041721 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:28.041768 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:28.041782 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:28.041857 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:28.042136 (+ 279us) server_negotiation.cc:299] Negotiation successful
0809 05:22:28.042225 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260809 05:22:28.043996 4485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.030773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38343 (local address 127.22.1.65:45359)
0809 05:22:28.031974 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:28.032034 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:28.034249 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:28.035286 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:28.035297 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:28.035476 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:28.036332 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.036359 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.037479 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.037485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.039310 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.039326 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.040163 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.040170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.040466 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.041291 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:28.041294 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:28.041350 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:28.041477 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:28.041874 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:28.041884 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:28.041893 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 05:22:28.042082 (+ 189us) client_negotiation.cc:240] Negotiation successful
0809 05:22:28.042543 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":320,"threads_started":1}
I20260809 05:22:28.045775 4482 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38343
I20260809 05:22:28.046154 4482 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:28.047008 4482 heartbeater.cc:511] Master 127.22.1.126:38343 requested a full tablet report, sending...
I20260809 05:22:28.049381 4180 ts_manager.cc:194] Registered new tserver with Master: 76fd95205b3f44c48032b19a9352dea7 (127.22.1.65:45227)
I20260809 05:22:28.051028 4180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.65:45359
W20260809 05:22:28.388417 4487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:28.388751 4487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:28.388875 4487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:28.397375 4487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:28.397502 4487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:28.397650 4487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:28.407155 4487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:28.408764 4487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:28.410768 4487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:28.424868 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:28.424793 4504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:28.424839 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:28.425295 4487 server_base.cc:1034] running on GCE node
I20260809 05:22:28.426154 4487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:28.427217 4487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:28.428503 4487 hybrid_clock.cc:648] HybridClock initialized: now 1786252948428413 us; error 50 us; skew 500 ppm
I20260809 05:22:28.428844 4487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:28.431339 4487 webserver.cc:460] Webserver started at http://127.22.1.66:32963/ using document root <none> and password file <none>
I20260809 05:22:28.432080 4487 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:28.432194 4487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:28.432530 4487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:28.435632 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "29068855008e46e6a8311858ada74c82"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "ce0bb5bde721566983be03703965d44e"
server_key_iv: "3a15a2fd708933d41eca432665ec0b2a"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.436492 4487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "29068855008e46e6a8311858ada74c82"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "ce0bb5bde721566983be03703965d44e"
server_key_iv: "3a15a2fd708933d41eca432665ec0b2a"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.442291 4487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:22:28.446345 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.447925 4487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:22:28.448127 4487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "29068855008e46e6a8311858ada74c82"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "ce0bb5bde721566983be03703965d44e"
server_key_iv: "3a15a2fd708933d41eca432665ec0b2a"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.448364 4487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:28.462543 4487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:28.463465 4487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:28.463788 4487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:28.464773 4487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:28.466403 4487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:28.466493 4487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.466586 4487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:28.466640 4487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.497507 4487 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:35349
I20260809 05:22:28.497571 4677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:35349 every 8 connection(s)
I20260809 05:22:28.498836 4487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:28.504961 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 4487
I20260809 05:22:28.505200 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:22:28.505759 22533 external_mini_cluster.cc:1428] Setting key e4219f97cd0b7c43a994295a134ffe64
I20260809 05:22:28.509092 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--builtin_ntp_servers=127.22.1.84:35209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260809 05:22:28.513128 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.501482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:38187 (local address 127.22.1.126:38343)
0809 05:22:28.501658 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:28.501663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:28.503113 (+ 1450us) server_negotiation.cc:407] Connection header received
0809 05:22:28.505319 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:28.505326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:28.505416 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:28.505506 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:28.506949 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.508028 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.510296 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.510704 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.510852 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.512413 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:28.512418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:28.512422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:28.512442 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:28.512476 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:28.512489 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:28.512561 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:28.512801 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 05:22:28.512885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:22:28.514665 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.501939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38343 (local address 127.22.1.66:38187)
0809 05:22:28.502906 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:28.502970 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:28.504895 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:28.505801 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:28.505815 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:28.505946 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:28.506752 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.506778 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.508191 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.508197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.510154 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.510170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.510964 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.510971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.511263 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.512028 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:28.512031 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:28.512088 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:28.512198 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:28.512581 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:28.512591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:28.512599 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 05:22:28.512799 (+ 200us) client_negotiation.cc:240] Negotiation successful
0809 05:22:28.513282 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":302,"threads_started":1}
I20260809 05:22:28.516252 4678 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38343
I20260809 05:22:28.516605 4678 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:28.517606 4678 heartbeater.cc:511] Master 127.22.1.126:38343 requested a full tablet report, sending...
I20260809 05:22:28.519147 4180 ts_manager.cc:194] Registered new tserver with Master: 29068855008e46e6a8311858ada74c82 (127.22.1.66:35349)
I20260809 05:22:28.519948 4180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.66:38187
W20260809 05:22:28.855392 4683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:28.855728 4683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:28.855861 4683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:28.863431 4683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:28.863569 4683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:28.863719 4683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:28.873466 4683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:28.874838 4683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:28.876637 4683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:28.889546 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:28.889583 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:28.889621 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:28.890265 4683 server_base.cc:1034] running on GCE node
I20260809 05:22:28.890908 4683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:28.891911 4683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:28.893189 4683 hybrid_clock.cc:648] HybridClock initialized: now 1786252948893110 us; error 43 us; skew 500 ppm
I20260809 05:22:28.893580 4683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:28.896057 4683 webserver.cc:460] Webserver started at http://127.22.1.67:42585/ using document root <none> and password file <none>
I20260809 05:22:28.896791 4683 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:28.896900 4683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:28.897276 4683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:28.900242 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "55cd40b815a54811934aed4903e64539"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "e23e158eac9c579a7b257e7863f3f395"
server_key_iv: "04db25db1252d097c16c38760532c24e"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.901059 4683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "55cd40b815a54811934aed4903e64539"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "e23e158eac9c579a7b257e7863f3f395"
server_key_iv: "04db25db1252d097c16c38760532c24e"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.906546 4683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:22:28.910305 4705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.911759 4683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:22:28.911937 4683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "55cd40b815a54811934aed4903e64539"
format_stamp: "Formatted at 2026-08-09 05:22:28 on dist-test-slave-rmhg"
server_key: "e23e158eac9c579a7b257e7863f3f395"
server_key_iv: "04db25db1252d097c16c38760532c24e"
server_key_version: "encryptionkey@0"
I20260809 05:22:28.912160 4683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:28.936956 4683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:28.937963 4683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:28.938246 4683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:28.939265 4683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:28.940800 4683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:28.940899 4683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.941004 4683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:28.941080 4683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:28.972327 4683 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:42049
I20260809 05:22:28.972400 4886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:42049 every 8 connection(s)
I20260809 05:22:28.973726 4683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:28.977541 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 4683
I20260809 05:22:28.977757 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:22:28.978329 22533 external_mini_cluster.cc:1428] Setting key c8143fa486b67db0510f545249d9d9bf
I20260809 05:22:28.987102 4484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.976372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:53391 (local address 127.22.1.126:38343)
0809 05:22:28.976593 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:28.976600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:28.978407 (+ 1807us) server_negotiation.cc:407] Connection header received
0809 05:22:28.980585 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:28.980590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:28.980681 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:28.980762 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:28.982228 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.982974 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.984551 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.984943 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.985048 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.986406 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:28.986411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:28.986414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:28.986433 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:28.986466 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:28.986478 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:28.986550 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:28.986745 (+ 195us) server_negotiation.cc:299] Negotiation successful
0809 05:22:28.986829 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:22:28.988122 4892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:28.976942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38343 (local address 127.22.1.67:53391)
0809 05:22:28.978205 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:28.978255 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:28.980304 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:28.981064 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:28.981076 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:28.981251 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:28.982071 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.982093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.983116 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.983120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.984407 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:28.984420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:28.985189 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:28.985198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:28.985424 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:28.986074 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:28.986077 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:28.986131 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:28.986223 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:28.986553 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:28.986559 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:28.986564 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 05:22:28.986724 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 05:22:28.987059 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":351,"threads_started":1}
I20260809 05:22:28.989601 4888 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38343
I20260809 05:22:28.989920 4888 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:28.990703 4888 heartbeater.cc:511] Master 127.22.1.126:38343 requested a full tablet report, sending...
I20260809 05:22:28.992156 4180 ts_manager.cc:194] Registered new tserver with Master: 55cd40b815a54811934aed4903e64539 (127.22.1.67:42049)
I20260809 05:22:28.993199 4180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.67:53391
I20260809 05:22:28.994360 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 05:22:29.000722 4484 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:28.999181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.22.1.126:38343)
0809 05:22:28.999439 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:28.999446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:28.999775 (+ 329us) server_negotiation.cc:407] Connection header received
0809 05:22:28.999918 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:28.999922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:28.999992 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:29.000075 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:29.000437 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46976: BlockingRecv error: recv got EOF from 127.0.0.1:46976 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:22:29.001752 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 4277
I20260809 05:22:29.009598 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:29.002631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.22.1.65:45227)
0809 05:22:29.003179 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:29.003186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:29.003211 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:22:29.003268 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:29.003274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:29.003493 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:29.003650 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:29.004068 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.005097 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.006370 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.006963 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.007090 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:29.008105 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:29.008113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:29.008128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:29.008750 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:29.008801 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:29.008821 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:29.008946 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:29.009083 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 05:22:29.009249 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260809 05:22:29.010818 4476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:29.091327 4482 heartbeater.cc:503] Master 127.22.1.126:38343 was elected leader, sending a full tablet report...
I20260809 05:22:29.093359 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 4277
I20260809 05:22:29.116511 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 4487
I20260809 05:22:29.124346 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:29.117554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.22.1.66:35349)
0809 05:22:29.118059 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:29.118066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:29.118095 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:22:29.118159 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:29.118166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:29.118383 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:29.118514 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:29.118907 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.119916 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.121278 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.121849 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.121948 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:29.122987 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:29.122994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:29.123017 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:29.123636 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:29.123681 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:29.123706 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:29.123794 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:29.123901 (+ 107us) server_negotiation.cc:299] Negotiation successful
0809 05:22:29.124046 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260809 05:22:29.125690 4671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:29.206382 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 4487
I20260809 05:22:29.227496 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 4683
I20260809 05:22:29.235965 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:29.228548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.22.1.67:42049)
0809 05:22:29.229299 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:29.229306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:29.229339 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:22:29.229401 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:29.229407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:29.229686 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:29.229821 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:29.230274 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.231364 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.232719 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:29.233382 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:29.233496 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:29.234526 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:29.234535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:29.234551 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:29.235164 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:29.235221 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:29.235243 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:29.235333 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:29.235448 (+ 115us) server_negotiation.cc:299] Negotiation successful
0809 05:22:29.235599 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":402,"threads_started":1}
I20260809 05:22:29.237329 4882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:29.318898 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 4683
I20260809 05:22:29.342490 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 4132
I20260809 05:22:29.343770 4231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:29.447968 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 4132
2026-08-09T05:22:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2600 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-09T05:22:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:29Z Disabled control of system clock
I20260809 05:22:29.532460 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44923
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:44923
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 05:22:29.876572 4922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:29.876888 4922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:29.877000 4922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:29.884362 4922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:29.884514 4922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:29.884563 4922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:29.884646 4922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:29.884701 4922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:29.894217 4922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:44923
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:29.895591 4922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:29.897663 4922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:29.909655 4934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:29.909865 4938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:29.909711 4935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:29.910671 4922 server_base.cc:1034] running on GCE node
I20260809 05:22:29.911458 4922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:29.912916 4922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:29.914155 4922 hybrid_clock.cc:648] HybridClock initialized: now 1786252949914079 us; error 54 us; skew 500 ppm
I20260809 05:22:29.914510 4922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:29.916718 4922 webserver.cc:460] Webserver started at http://127.22.1.126:34601/ using document root <none> and password file <none>
I20260809 05:22:29.917495 4922 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:29.917587 4922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:29.917870 4922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:29.920878 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "48d7d64df4ed4121889c5ad711de4b3a"
format_stamp: "Formatted at 2026-08-09 05:22:29 on dist-test-slave-rmhg"
server_key: "4d844c78ae17bdc6f394cfb2a4f6ecc0"
server_key_iv: "3967c12f8d1721db1f0001b665c55e7a"
server_key_version: "encryptionkey@0"
I20260809 05:22:29.921730 4922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "48d7d64df4ed4121889c5ad711de4b3a"
format_stamp: "Formatted at 2026-08-09 05:22:29 on dist-test-slave-rmhg"
server_key: "4d844c78ae17bdc6f394cfb2a4f6ecc0"
server_key_iv: "3967c12f8d1721db1f0001b665c55e7a"
server_key_version: "encryptionkey@0"
I20260809 05:22:29.927112 4922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:22:29.930761 4947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:29.932148 4922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:22:29.932307 4922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "48d7d64df4ed4121889c5ad711de4b3a"
format_stamp: "Formatted at 2026-08-09 05:22:29 on dist-test-slave-rmhg"
server_key: "4d844c78ae17bdc6f394cfb2a4f6ecc0"
server_key_iv: "3967c12f8d1721db1f0001b665c55e7a"
server_key_version: "encryptionkey@0"
I20260809 05:22:29.932508 4922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:29.938287 4922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:29.939055 4922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:29.939329 4922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:29.956310 4922 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:44923
I20260809 05:22:29.956338 5024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:44923 every 8 connection(s)
I20260809 05:22:29.957738 4922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:22:29.958159 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 4922
I20260809 05:22:29.958362 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:22:29.959254 22533 external_mini_cluster.cc:1428] Setting key 67ae6652843d97ecd9bee5988edcc6ea
I20260809 05:22:29.962723 5025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:29.968009 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:29.961572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.22.1.126:44923)
0809 05:22:29.962610 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:29.962622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:29.962670 (+ 48us) server_negotiation.cc:407] Connection header received
0809 05:22:29.964324 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:29.964348 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:29.964798 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:29.965211 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:29.966286 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:29.966292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:29.966309 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:29.966331 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:29.966411 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:29.966427 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:29.966656 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:29.966803 (+ 147us) server_negotiation.cc:299] Negotiation successful
0809 05:22:29.967068 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":359,"threads_started":1}
I20260809 05:22:29.974265 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a: Bootstrap starting.
I20260809 05:22:29.977795 5025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:29.978879 5025 log.cc:824] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:29.981891 5025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a: No bootstrap required, opened a new log
I20260809 05:22:29.989387 5025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } }
I20260809 05:22:29.989813 5025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:29.989895 5025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d7d64df4ed4121889c5ad711de4b3a, State: Initialized, Role: FOLLOWER
I20260809 05:22:29.990515 5025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } }
I20260809 05:22:29.990751 5025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:22:29.990856 5025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:22:29.990984 5025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:29.993280 5025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } }
I20260809 05:22:29.993780 5025 leader_election.cc:302] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48d7d64df4ed4121889c5ad711de4b3a; no voters:
I20260809 05:22:29.994383 5025 leader_election.cc:288] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:22:29.994544 5034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:29.995262 5034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [term 1 LEADER]: Becoming Leader. State: Replica: 48d7d64df4ed4121889c5ad711de4b3a, State: Running, Role: LEADER
I20260809 05:22:29.995805 5034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } }
I20260809 05:22:29.996484 5025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:22:29.998661 5036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48d7d64df4ed4121889c5ad711de4b3a. Latest consensus state: current_term: 1 leader_uuid: "48d7d64df4ed4121889c5ad711de4b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } } }
I20260809 05:22:29.999078 5036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:29.999866 5035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48d7d64df4ed4121889c5ad711de4b3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48d7d64df4ed4121889c5ad711de4b3a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44923 } } }
I20260809 05:22:30.000205 5035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:30.001863 5049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:22:30.007572 5049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:22:30.016703 5049 catalog_manager.cc:1347] Generated new cluster ID: 73916c9885574eb3a17af797a03c0a1c
I20260809 05:22:30.016836 5049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:22:30.041585 5049 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:22:30.042120 5049 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:22:30.056519 5049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48d7d64df4ed4121889c5ad711de4b3a: Generated new TSK 0
I20260809 05:22:30.057672 5049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:22:30.077061 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--builtin_ntp_servers=127.22.1.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260809 05:22:30.437469 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:30.437788 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:30.437920 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:30.445423 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:30.445538 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:30.445693 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:30.455075 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:30.456430 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:30.458328 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:30.471521 5078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:30.471448 5081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:30.471526 5079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:30.472393 5069 server_base.cc:1034] running on GCE node
I20260809 05:22:30.473093 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:30.474123 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:30.475344 5069 hybrid_clock.cc:648] HybridClock initialized: now 1786252950475286 us; error 45 us; skew 500 ppm
I20260809 05:22:30.475714 5069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:30.478166 5069 webserver.cc:460] Webserver started at http://127.22.1.65:46497/ using document root <none> and password file <none>
I20260809 05:22:30.478909 5069 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:30.479007 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:30.479348 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:30.484961 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "a9fdb274f7a54c5994990ce5b915d2e9"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "a40fd1afa744acac20ff3568009ded33"
server_key_iv: "5493bf7cde7372eda36d9f849780a48d"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.485961 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "a9fdb274f7a54c5994990ce5b915d2e9"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "a40fd1afa744acac20ff3568009ded33"
server_key_iv: "5493bf7cde7372eda36d9f849780a48d"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.491766 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260809 05:22:30.495564 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.497135 5069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 05:22:30.497345 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "a9fdb274f7a54c5994990ce5b915d2e9"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "a40fd1afa744acac20ff3568009ded33"
server_key_iv: "5493bf7cde7372eda36d9f849780a48d"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.497605 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:30.503655 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:30.504536 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:30.504801 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:30.505875 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:30.507452 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:30.507565 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.507679 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:30.507748 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.539155 5069 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:42885
I20260809 05:22:30.539202 5257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:42885 every 8 connection(s)
I20260809 05:22:30.540493 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:30.548754 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 5069
I20260809 05:22:30.548964 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:22:30.548980 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:30.543112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:52041 (local address 127.22.1.126:44923)
0809 05:22:30.543575 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:30.543580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:30.544725 (+ 1145us) server_negotiation.cc:407] Connection header received
0809 05:22:30.546530 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:30.546535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:30.546606 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:30.546707 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:30.548103 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:30.548110 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:30.548114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:30.548134 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:30.548189 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:30.548205 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:30.548345 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:30.548578 (+ 233us) server_negotiation.cc:299] Negotiation successful
0809 05:22:30.548654 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":235,"threads_started":1}
I20260809 05:22:30.549693 22533 external_mini_cluster.cc:1428] Setting key 8e25fb858d6e86860ad51f422ab7c719
I20260809 05:22:30.549983 5265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:30.543627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44923 (local address 127.22.1.65:52041)
0809 05:22:30.544559 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:30.544599 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:30.546233 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:30.546992 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:30.547000 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:30.547103 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:30.547781 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:30.547783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:30.547824 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:30.547914 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:30.548345 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:30.548355 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:30.548362 (+ 7us) client_negotiation.cc:769] Sending connection context
0809 05:22:30.548636 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 05:22:30.548863 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
I20260809 05:22:30.551563 5261 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44923
I20260809 05:22:30.551896 5261 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:30.552505 5261 heartbeater.cc:511] Master 127.22.1.126:44923 requested a full tablet report, sending...
I20260809 05:22:30.552683 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--builtin_ntp_servers=127.22.1.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 05:22:30.554646 4976 ts_manager.cc:194] Registered new tserver with Master: a9fdb274f7a54c5994990ce5b915d2e9 (127.22.1.65:42885)
W20260809 05:22:30.893302 5266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:30.893600 5266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:30.893715 5266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:30.901000 5266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:30.901122 5266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:30.901294 5266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:30.910905 5266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:30.912277 5266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:30.914145 5266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:30.927314 5274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:30.927350 5272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:30.927388 5271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:30.928519 5266 server_base.cc:1034] running on GCE node
I20260809 05:22:30.929207 5266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:30.930223 5266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:30.931420 5266 hybrid_clock.cc:648] HybridClock initialized: now 1786252950931362 us; error 50 us; skew 500 ppm
I20260809 05:22:30.931771 5266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:30.934276 5266 webserver.cc:460] Webserver started at http://127.22.1.66:40555/ using document root <none> and password file <none>
I20260809 05:22:30.935019 5266 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:30.935127 5266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:30.935459 5266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:30.938668 5266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "58b5961b845b4ab0a02d3b21ffcd6c6b"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "009b8f035550f6092f4233556bda344c"
server_key_iv: "2135c5c0acec2ed64938239adf3e9ab0"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.939469 5266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "58b5961b845b4ab0a02d3b21ffcd6c6b"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "009b8f035550f6092f4233556bda344c"
server_key_iv: "2135c5c0acec2ed64938239adf3e9ab0"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.944975 5266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 05:22:30.948553 5283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.950201 5266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:30.950369 5266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "58b5961b845b4ab0a02d3b21ffcd6c6b"
format_stamp: "Formatted at 2026-08-09 05:22:30 on dist-test-slave-rmhg"
server_key: "009b8f035550f6092f4233556bda344c"
server_key_iv: "2135c5c0acec2ed64938239adf3e9ab0"
server_key_version: "encryptionkey@0"
I20260809 05:22:30.950623 5266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:30.956650 5266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:30.957682 5266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:30.957984 5266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:30.959059 5266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:30.960820 5266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:30.960923 5266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.961027 5266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:30.961102 5266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:30.999827 5266 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:39733
I20260809 05:22:30.999866 5448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:39733 every 8 connection(s)
I20260809 05:22:31.001420 5266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:31.009649 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.003999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:38333 (local address 127.22.1.126:44923)
0809 05:22:31.004209 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.004214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.005717 (+ 1503us) server_negotiation.cc:407] Connection header received
0809 05:22:31.007433 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.007437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.007493 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.007591 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.008983 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:31.008988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:31.008992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:31.009009 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:31.009044 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:31.009056 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:31.009189 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:31.009357 (+ 168us) server_negotiation.cc:299] Negotiation successful
0809 05:22:31.009415 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 05:22:31.010349 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 5266
I20260809 05:22:31.010527 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:22:31.010637 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.004578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44923 (local address 127.22.1.66:38333)
0809 05:22:31.005570 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:31.005608 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:31.007154 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:31.007804 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:31.007813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:31.007914 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:31.008595 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:31.008598 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:31.008652 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:31.008765 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:31.009139 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:31.009145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:31.009171 (+ 26us) client_negotiation.cc:769] Sending connection context
0809 05:22:31.009403 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 05:22:31.009614 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":267,"threads_started":1}
I20260809 05:22:31.011023 22533 external_mini_cluster.cc:1428] Setting key 2ab1a5297f7adc230568197f41f01e66
I20260809 05:22:31.012049 5449 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44923
I20260809 05:22:31.012379 5449 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:31.013123 5449 heartbeater.cc:511] Master 127.22.1.126:44923 requested a full tablet report, sending...
I20260809 05:22:31.013926 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--builtin_ntp_servers=127.22.1.84:41117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260809 05:22:31.014663 4976 ts_manager.cc:194] Registered new tserver with Master: 58b5961b845b4ab0a02d3b21ffcd6c6b (127.22.1.66:39733)
W20260809 05:22:31.363821 5456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:31.364177 5456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:31.364289 5456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:31.375094 5456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:31.375247 5456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:31.375442 5456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:31.389243 5456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:31.391112 5456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:31.393597 5456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:31.408951 5466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:31.409312 5465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:31.408932 5471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:31.409651 5456 server_base.cc:1034] running on GCE node
I20260809 05:22:31.410276 5456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:31.411227 5456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:31.412441 5456 hybrid_clock.cc:648] HybridClock initialized: now 1786252951412397 us; error 34 us; skew 500 ppm
I20260809 05:22:31.412770 5456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:22:31.415076 5456 webserver.cc:460] Webserver started at http://127.22.1.67:35735/ using document root <none> and password file <none>
I20260809 05:22:31.415786 5456 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:31.415889 5456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:31.416221 5456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:31.419203 5456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "fc7da00397c44a37ac24056b45be2165"
format_stamp: "Formatted at 2026-08-09 05:22:31 on dist-test-slave-rmhg"
server_key: "1303511e720295aaf44428e224b0e580"
server_key_iv: "3e6f1c75419f73e86b18d26a5940dcb0"
server_key_version: "encryptionkey@0"
I20260809 05:22:31.420029 5456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "fc7da00397c44a37ac24056b45be2165"
format_stamp: "Formatted at 2026-08-09 05:22:31 on dist-test-slave-rmhg"
server_key: "1303511e720295aaf44428e224b0e580"
server_key_iv: "3e6f1c75419f73e86b18d26a5940dcb0"
server_key_version: "encryptionkey@0"
I20260809 05:22:31.425418 5456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:22:31.428915 5478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:31.430405 5456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:22:31.430572 5456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "fc7da00397c44a37ac24056b45be2165"
format_stamp: "Formatted at 2026-08-09 05:22:31 on dist-test-slave-rmhg"
server_key: "1303511e720295aaf44428e224b0e580"
server_key_iv: "3e6f1c75419f73e86b18d26a5940dcb0"
server_key_version: "encryptionkey@0"
I20260809 05:22:31.430783 5456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:31.436691 5456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:31.437582 5456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:31.437839 5456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:31.438776 5456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:31.440213 5456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:31.440310 5456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:31.440423 5456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:31.440492 5456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:31.471323 5456 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:37463
I20260809 05:22:31.471380 5655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:37463 every 8 connection(s)
I20260809 05:22:31.472658 5456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:31.480921 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.475304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:50693 (local address 127.22.1.126:44923)
0809 05:22:31.475484 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.475490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.476936 (+ 1446us) server_negotiation.cc:407] Connection header received
0809 05:22:31.478644 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.478648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.478726 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.478818 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.480216 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:31.480221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:31.480224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:31.480242 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:31.480285 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:31.480298 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:31.480399 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:31.480544 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 05:22:31.480600 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:22:31.481103 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 5456
I20260809 05:22:31.481299 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:22:31.481746 22533 external_mini_cluster.cc:1428] Setting key 39297b345828bf80de6e02c80e9acfaa
I20260809 05:22:31.481961 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.475842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44923 (local address 127.22.1.67:50693)
0809 05:22:31.476771 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:31.476817 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:31.478383 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:31.479104 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:31.479114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:31.479217 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:31.479874 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:22:31.479877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:31.479919 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:22:31.480037 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:31.480378 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:31.480385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:31.480391 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:22:31.480623 (+ 232us) client_negotiation.cc:240] Negotiation successful
0809 05:22:31.480846 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":263,"threads_started":1}
I20260809 05:22:31.483539 5656 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44923
I20260809 05:22:31.483871 5656 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:31.484666 5656 heartbeater.cc:511] Master 127.22.1.126:44923 requested a full tablet report, sending...
I20260809 05:22:31.486229 4976 ts_manager.cc:194] Registered new tserver with Master: fc7da00397c44a37ac24056b45be2165 (127.22.1.67:37463)
I20260809 05:22:31.497869 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260809 05:22:31.504311 5264 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:22:31.502559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.22.1.126:44923)
0809 05:22:31.502756 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.502761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.503297 (+ 536us) server_negotiation.cc:407] Connection header received
0809 05:22:31.503498 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.503502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.503560 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.503644 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.504034 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57346: BlockingRecv error: recv got EOF from 127.0.0.1:57346 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:22:31.505290 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 5069
I20260809 05:22:31.508699 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.506217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.22.1.65:42885)
0809 05:22:31.506704 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.506711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.506740 (+ 29us) server_negotiation.cc:407] Connection header received
0809 05:22:31.506801 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.506807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.507024 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.507158 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.508035 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:31.508042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:31.508056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:31.508068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:31.508111 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:31.508125 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:31.508245 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:31.508378 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 05:22:31.508472 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260809 05:22:31.510021 5251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:31.603148 5261 heartbeater.cc:503] Master 127.22.1.126:44923 was elected leader, sending a full tablet report...
I20260809 05:22:31.606467 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 5069
I20260809 05:22:31.635288 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 5266
I20260809 05:22:31.641304 5672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.636841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.22.1.66:39733)
0809 05:22:31.637752 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.637762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.637810 (+ 48us) server_negotiation.cc:407] Connection header received
0809 05:22:31.637924 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.637932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.638371 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.638587 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.640115 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:31.640132 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:31.640155 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:31.640202 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:31.640281 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:31.640308 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:31.640487 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:31.640659 (+ 172us) server_negotiation.cc:299] Negotiation successful
0809 05:22:31.640809 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":422,"threads_started":1}
I20260809 05:22:31.643493 5444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:31.763500 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 5266
I20260809 05:22:31.795297 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 5456
I20260809 05:22:31.802412 5677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:31.796926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.22.1.67:37463)
0809 05:22:31.798051 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:31.798069 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:31.798173 (+ 104us) server_negotiation.cc:407] Connection header received
0809 05:22:31.798280 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:31.798287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:31.798701 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:31.798920 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:31.800906 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:31.800922 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:31.800944 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:22:31.800973 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:31.801050 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:22:31.801078 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:31.801434 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:31.801650 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 05:22:31.801821 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":622,"threads_started":1}
I20260809 05:22:31.804550 5647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:31.921044 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 5456
I20260809 05:22:31.952970 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 4922
I20260809 05:22:31.954926 5020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:32.103291 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 4922
2026-08-09T05:22:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2673 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:22:32 dist-test-slave-rmhg krb5kdc[5685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:22:32 dist-test-slave-rmhg krb5kdc[5685](info): set up 2 sockets
Aug 09 05:22:32 dist-test-slave-rmhg krb5kdc[5685](info): commencing operation
krb5kdc: starting...
W20260809 05:22:34.058663 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.887s user 0.003s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:22:34 dist-test-slave-rmhg krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:22:34.518066 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38411
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:34.905582 5707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:34.905905 5707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:34.906050 5707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:34.913595 5707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:22:34.913738 5707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:34.913816 5707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:34.913898 5707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:22:34.913957 5707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:22:34.924285 5707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:34.925809 5707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:34.927312 5707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:34.939399 5718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:34.939366 5715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:34.939443 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:34.942559 5707 server_base.cc:1034] running on GCE node
I20260809 05:22:34.943408 5707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:34.944955 5707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:34.946175 5707 hybrid_clock.cc:648] HybridClock initialized: now 1786252954946100 us; error 66 us; skew 500 ppm
I20260809 05:22:34.946517 5707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:34 dist-test-slave-rmhg krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:34.954161 5707 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:22:34.955829 5707 webserver.cc:460] Webserver started at http://127.22.1.126:44407/ using document root <none> and password file <none>
I20260809 05:22:34.956598 5707 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:34.956724 5707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:34.957048 5707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:34.960192 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "7b19fe625e404dbb827cab687ba7833e"
format_stamp: "Formatted at 2026-08-09 05:22:34 on dist-test-slave-rmhg"
server_key: "77f95e95417f047a1bf606e9b32faed6"
server_key_iv: "4da2165b3af9288739eca55c33ea443e"
server_key_version: "encryptionkey@0"
I20260809 05:22:34.960953 5707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "7b19fe625e404dbb827cab687ba7833e"
format_stamp: "Formatted at 2026-08-09 05:22:34 on dist-test-slave-rmhg"
server_key: "77f95e95417f047a1bf606e9b32faed6"
server_key_iv: "4da2165b3af9288739eca55c33ea443e"
server_key_version: "encryptionkey@0"
I20260809 05:22:34.966248 5707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:22:34.969722 5729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:34.971035 5707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:22:34.971199 5707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "7b19fe625e404dbb827cab687ba7833e"
format_stamp: "Formatted at 2026-08-09 05:22:34 on dist-test-slave-rmhg"
server_key: "77f95e95417f047a1bf606e9b32faed6"
server_key_iv: "4da2165b3af9288739eca55c33ea443e"
server_key_version: "encryptionkey@0"
I20260809 05:22:34.971374 5707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:34.994588 5707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:34.998724 5707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:34.999002 5707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:35.017361 5707 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38411
I20260809 05:22:35.017374 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38411 every 8 connection(s)
I20260809 05:22:35.018819 5707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:22:35.022534 5816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:35.028893 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 5707
I20260809 05:22:35.029100 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:22:35.029829 22533 external_mini_cluster.cc:1428] Setting key 5dd374bf6b552e5031dc2cc3990584fc
I20260809 05:22:35.033519 5816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: Bootstrap starting.
I20260809 05:22:35.037305 5816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:35.038563 5816 log.cc:824] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:35.042147 5816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: No bootstrap required, opened a new log
Aug 09 05:22:35 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:35.050879 5816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } }
I20260809 05:22:35.051364 5816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:35.051477 5816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b19fe625e404dbb827cab687ba7833e, State: Initialized, Role: FOLLOWER
I20260809 05:22:35.052202 5816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } }
I20260809 05:22:35.052520 5816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:22:35.052726 5816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:22:35.052929 5816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:35.054111 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:35.031932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.22.1.126:38411)
0809 05:22:35.032736 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:35.032749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:35.032792 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:22:35.034619 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:35.034646 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:35.035099 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:35.035458 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:35.037470 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.038901 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.040703 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.041458 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.041608 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:35.046176 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:35.046187 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:35.046207 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:35.046260 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:35.049544 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:35.050400 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:35.050407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:35.050414 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:35.050589 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:35.051032 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:35.051035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:35.051037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:35.051691 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:35.052029 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:35.052389 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 05:22:35.052875 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":354,"threads_started":1}
I20260809 05:22:35.055900 5816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } }
I20260809 05:22:35.056416 5816 leader_election.cc:302] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b19fe625e404dbb827cab687ba7833e; no voters:
I20260809 05:22:35.057078 5816 leader_election.cc:288] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:22:35.057227 5824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:35.058094 5824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [term 1 LEADER]: Becoming Leader. State: Replica: 7b19fe625e404dbb827cab687ba7833e, State: Running, Role: LEADER
I20260809 05:22:35.058609 5824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } }
I20260809 05:22:35.059209 5816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:22:35.061506 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b19fe625e404dbb827cab687ba7833e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } } }
I20260809 05:22:35.061892 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:35.061570 5827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b19fe625e404dbb827cab687ba7833e. Latest consensus state: current_term: 1 leader_uuid: "7b19fe625e404dbb827cab687ba7833e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b19fe625e404dbb827cab687ba7833e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38411 } } }
I20260809 05:22:35.062011 5827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e [sys.catalog]: This master's current role is: LEADER
I20260809 05:22:35.063555 5835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:22:35.069756 5835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:22:35.078447 5835 catalog_manager.cc:1347] Generated new cluster ID: f16878d10fcc4dd6a1c0b646fc716b05
I20260809 05:22:35.078576 5835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:22:35.090135 5835 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:22:35.091470 5835 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:22:35.108273 5835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: Generated new TSK 0
I20260809 05:22:35.109193 5835 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:22:35.174798 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:35.518311 5858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:35.518623 5858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:35.518731 5858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:35.526113 5858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:35.526242 5858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:35.526364 5858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:22:35.536444 5858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:35.537886 5858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:35.539222 5858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:35.552596 5867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:35.552664 5868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:35.552690 5872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:35.553748 5858 server_base.cc:1034] running on GCE node
I20260809 05:22:35.554409 5858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:35.555431 5858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:35.556648 5858 hybrid_clock.cc:648] HybridClock initialized: now 1786252955556595 us; error 46 us; skew 500 ppm
I20260809 05:22:35.556972 5858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:35 dist-test-slave-rmhg krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:35.560114 5858 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:22:35.561637 5858 webserver.cc:460] Webserver started at http://127.22.1.65:43853/ using document root <none> and password file <none>
I20260809 05:22:35.562300 5858 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:35.562399 5858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:35.562668 5858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:35.565752 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "7fa703e88a17424f80f723ae952c5f32"
format_stamp: "Formatted at 2026-08-09 05:22:35 on dist-test-slave-rmhg"
server_key: "d11479b4ab125055593103e1ecb87307"
server_key_iv: "7400fec0e6f566353cd61691fc0ab6b8"
server_key_version: "encryptionkey@0"
I20260809 05:22:35.566499 5858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "7fa703e88a17424f80f723ae952c5f32"
format_stamp: "Formatted at 2026-08-09 05:22:35 on dist-test-slave-rmhg"
server_key: "d11479b4ab125055593103e1ecb87307"
server_key_iv: "7400fec0e6f566353cd61691fc0ab6b8"
server_key_version: "encryptionkey@0"
I20260809 05:22:35.571739 5858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:22:35.575371 5883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:35.576893 5858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:22:35.577075 5858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "7fa703e88a17424f80f723ae952c5f32"
format_stamp: "Formatted at 2026-08-09 05:22:35 on dist-test-slave-rmhg"
server_key: "d11479b4ab125055593103e1ecb87307"
server_key_iv: "7400fec0e6f566353cd61691fc0ab6b8"
server_key_version: "encryptionkey@0"
I20260809 05:22:35.577311 5858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:35.617285 5858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:35.621696 5858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:35.622004 5858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:35.623005 5858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:35.624414 5858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:35.624562 5858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:35.624684 5858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:35.624740 5858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:35.655676 5858 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:34491
I20260809 05:22:35.655733 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:34491 every 8 connection(s)
I20260809 05:22:35.657027 5858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:22:35.662715 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 5858
I20260809 05:22:35.662925 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:22:35.663754 22533 external_mini_cluster.cc:1428] Setting key fb3e539e81387a7f731b29cbc692592d
Aug 09 05:22:35 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252955, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:35.679773 6057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:35.659756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:44503 (local address 127.22.1.126:38411)
0809 05:22:35.660255 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:35.660259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:35.661362 (+ 1103us) server_negotiation.cc:407] Connection header received
0809 05:22:35.663218 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:35.663225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:35.663317 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:35.663452 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:35.665237 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.666234 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.668600 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.669033 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.669219 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:35.673436 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:35.673442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:35.673446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:35.673481 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:35.676192 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:35.677311 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:35.677317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:35.677320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:35.677496 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:35.677896 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:35.677900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:35.677903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:35.678256 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:35.678412 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:35.679132 (+ 720us) server_negotiation.cc:299] Negotiation successful
0809 05:22:35.679376 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":247,"threads_started":1}
I20260809 05:22:35.680608 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:35.660419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38411 (local address 127.22.1.65:44503)
0809 05:22:35.661203 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:35.661244 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:35.662892 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:35.663649 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:35.663658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:35.664246 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:35.665029 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:35.665054 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.666433 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.666439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:35.668405 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:35.668421 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:35.669311 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:35.669318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:35.669619 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:35.670401 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:35.670430 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:35.673178 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:35.676342 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:35.676352 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:35.676371 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:35.676975 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:35.677600 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:35.677603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:35.677606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:35.677795 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:35.678370 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:35.678376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:35.678738 (+ 362us) client_negotiation.cc:769] Sending connection context
0809 05:22:35.678979 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 05:22:35.679402 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":286,"threads_started":1}
I20260809 05:22:35.682315 6055 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38411
I20260809 05:22:35.682642 6055 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:35.683408 6055 heartbeater.cc:511] Master 127.22.1.126:38411 requested a full tablet report, sending...
I20260809 05:22:35.686074 5760 ts_manager.cc:194] Registered new tserver with Master: 7fa703e88a17424f80f723ae952c5f32 (127.22.1.65:34491)
I20260809 05:22:35.687610 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:44503
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:22:35.725206 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:36.063187 6064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:36.063467 6064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:36.063596 6064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:36.070858 6064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:36.070976 6064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:36.071138 6064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:22:36.080891 6064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:36.082297 6064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:36.083652 6064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:36.097095 6070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:36.097124 6071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:36.097076 6073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:36.097807 6064 server_base.cc:1034] running on GCE node
I20260809 05:22:36.098448 6064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:36.099406 6064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:36.100625 6064 hybrid_clock.cc:648] HybridClock initialized: now 1786252956100563 us; error 57 us; skew 500 ppm
I20260809 05:22:36.100947 6064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:36 dist-test-slave-rmhg krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:36.104070 6064 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:22:36.105578 6064 webserver.cc:460] Webserver started at http://127.22.1.66:36473/ using document root <none> and password file <none>
I20260809 05:22:36.106261 6064 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:36.106377 6064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:36.106668 6064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:36.109692 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "55379cef0dd840c399c62010b3c01c47"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "702c4b527322c51e785c23631a68ddd2"
server_key_iv: "a3a750d0c4eb37703264fd578195b118"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.110455 6064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "55379cef0dd840c399c62010b3c01c47"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "702c4b527322c51e785c23631a68ddd2"
server_key_iv: "a3a750d0c4eb37703264fd578195b118"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.115723 6064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 05:22:36.119280 6083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.120649 6064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:22:36.120803 6064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "55379cef0dd840c399c62010b3c01c47"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "702c4b527322c51e785c23631a68ddd2"
server_key_iv: "a3a750d0c4eb37703264fd578195b118"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.120990 6064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:36.147399 6064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:36.151885 6064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:36.152185 6064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:36.153234 6064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:36.154603 6064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:36.154682 6064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.154798 6064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:36.154857 6064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.185606 6064 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:35987
I20260809 05:22:36.185652 6252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:35987 every 8 connection(s)
I20260809 05:22:36.186854 6064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:22:36.192569 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 6064
I20260809 05:22:36.192764 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:22:36.193380 22533 external_mini_cluster.cc:1428] Setting key 5a0661785908ef34527609493042f7f8
Aug 09 05:22:36 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:36.208796 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.189468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:44611 (local address 127.22.1.126:38411)
0809 05:22:36.189871 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.189876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.190952 (+ 1076us) server_negotiation.cc:407] Connection header received
0809 05:22:36.192978 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.192983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.193055 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.193176 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:36.195158 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.196015 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.198339 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.198782 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.198887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.203001 (+ 4114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:36.203007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:36.203010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:36.203045 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:36.205623 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.206489 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.206494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.206498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.206654 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.207042 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.207046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.207049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.207386 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:36.207532 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.208185 (+ 653us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.208407 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":192,"threads_started":1}
I20260809 05:22:36.209621 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.190049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38411 (local address 127.22.1.66:44611)
0809 05:22:36.190793 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:36.190839 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:36.192650 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:36.193466 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:36.193477 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:36.194173 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:36.194974 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.195000 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.196177 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.196183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.198172 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.198194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.199016 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.199023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.199296 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.200004 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:36.200034 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:36.202729 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:36.205765 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:36.205775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:36.205793 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:36.206364 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:36.206751 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:36.206755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:36.206757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:36.206941 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:36.207485 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:36.207490 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:36.207814 (+ 324us) client_negotiation.cc:769] Sending connection context
0809 05:22:36.208040 (+ 226us) client_negotiation.cc:240] Negotiation successful
0809 05:22:36.208431 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":287,"threads_started":1}
I20260809 05:22:36.211092 6254 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38411
I20260809 05:22:36.211431 6254 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:36.212301 6254 heartbeater.cc:511] Master 127.22.1.126:38411 requested a full tablet report, sending...
I20260809 05:22:36.213953 5760 ts_manager.cc:194] Registered new tserver with Master: 55379cef0dd840c399c62010b3c01c47 (127.22.1.66:35987)
I20260809 05:22:36.214756 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:44611
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:22:36.252573 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:22:36.597024 6264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:22:36.597365 6264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:22:36.597523 6264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:22:36.604811 6264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:22:36.604938 6264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:22:36.605098 6264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:22:36.615043 6264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:22:36.616508 6264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:22:36.617982 6264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:22:36.631265 6273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:36.631245 6274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:22:36.631317 6277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:22:36.632275 6264 server_base.cc:1034] running on GCE node
I20260809 05:22:36.632902 6264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:22:36.633904 6264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:22:36.635104 6264 hybrid_clock.cc:648] HybridClock initialized: now 1786252956635053 us; error 45 us; skew 500 ppm
I20260809 05:22:36.635412 6264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:22:36 dist-test-slave-rmhg krb5kdc[5685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:22:36.638491 6264 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:22:36.639967 6264 webserver.cc:460] Webserver started at http://127.22.1.67:45607/ using document root <none> and password file <none>
I20260809 05:22:36.640676 6264 fs_manager.cc:359] Metadata directory not provided
I20260809 05:22:36.640795 6264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:22:36.641086 6264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:22:36.644116 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "7286056c3ce74410a50bebfbf4d26e32"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "d139837f26630cc738e2b3c04a72f282"
server_key_iv: "9543c67d14a2df2da0f6814572c8f6fe"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.644922 6264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "7286056c3ce74410a50bebfbf4d26e32"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "d139837f26630cc738e2b3c04a72f282"
server_key_iv: "9543c67d14a2df2da0f6814572c8f6fe"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.650425 6264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260809 05:22:36.654090 6286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.655473 6264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:22:36.655637 6264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "7286056c3ce74410a50bebfbf4d26e32"
format_stamp: "Formatted at 2026-08-09 05:22:36 on dist-test-slave-rmhg"
server_key: "d139837f26630cc738e2b3c04a72f282"
server_key_iv: "9543c67d14a2df2da0f6814572c8f6fe"
server_key_version: "encryptionkey@0"
I20260809 05:22:36.655835 6264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:22:36.676667 6264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:22:36.680850 6264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:22:36.681143 6264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:22:36.682204 6264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:22:36.683624 6264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:22:36.683701 6264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.683822 6264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:22:36.683874 6264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:22:36.691596 6055 heartbeater.cc:503] Master 127.22.1.126:38411 was elected leader, sending a full tablet report...
I20260809 05:22:36.714670 6264 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:33427
I20260809 05:22:36.714718 6463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:33427 every 8 connection(s)
I20260809 05:22:36.715993 6264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:22:36.719341 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 6264
I20260809 05:22:36.719509 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:22:36.719982 22533 external_mini_cluster.cc:1428] Setting key fb13a9550c4926ed12c899ea6058d8a8
Aug 09 05:22:36 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252956, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:22:36.736570 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.718722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47105 (local address 127.22.1.126:38411)
0809 05:22:36.719293 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.719300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.720416 (+ 1116us) server_negotiation.cc:407] Connection header received
0809 05:22:36.722327 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.722332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.722436 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.722562 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:36.724172 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.725072 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.726773 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.727182 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.727288 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.730716 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:36.730721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:36.730725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:36.730769 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:36.733380 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.734193 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.734197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.734200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.734357 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.734853 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.734857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.734859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.735216 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:36.735363 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.736126 (+ 763us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.736297 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260809 05:22:36.737605 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.719338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38411 (local address 127.22.1.67:47105)
0809 05:22:36.720220 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:36.720270 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:36.722064 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:36.722840 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:36.722850 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:36.723404 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:22:36.724016 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.724034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.725241 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.725246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.726613 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.726627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.727418 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.727423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.727661 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.728280 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:22:36.728308 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:22:36.730497 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:22:36.733509 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:36.733522 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:36.733536 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:36.734053 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:36.734472 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:22:36.734476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:22:36.734480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:22:36.734739 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:22:36.735352 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:22:36.735361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:22:36.735773 (+ 412us) client_negotiation.cc:769] Sending connection context
0809 05:22:36.736001 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 05:22:36.736405 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":363,"threads_started":1}
I20260809 05:22:36.738996 6466 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38411
I20260809 05:22:36.739287 6466 heartbeater.cc:464] Registering TS with master...
I20260809 05:22:36.740025 6466 heartbeater.cc:511] Master 127.22.1.126:38411 requested a full tablet report, sending...
I20260809 05:22:36.741559 5760 ts_manager.cc:194] Registered new tserver with Master: 7286056c3ce74410a50bebfbf4d26e32 (127.22.1.67:33427)
I20260809 05:22:36.742364 5760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:47105
I20260809 05:22:36.747944 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:22:36.764703 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.752533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45032 (local address 127.22.1.126:38411)
0809 05:22:36.752708 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.752712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.753240 (+ 528us) server_negotiation.cc:407] Connection header received
0809 05:22:36.753371 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.753375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.753442 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.753552 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:36.754889 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.755665 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.756870 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.757340 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.757434 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.759605 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:36.759610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:36.759613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:36.759655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:36.762390 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.763014 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.763017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.763020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.763162 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:36.763446 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:36.763449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:36.763451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:36.763788 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:36.763981 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.764221 (+ 240us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.764399 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:22:36.768690 5760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:22:36.775614 5760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:22:36.803445 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.788237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33427 (local address 127.0.0.1:56896)
0809 05:22:36.789472 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:36.789512 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:36.789701 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:36.790791 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:36.790800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:36.790836 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:36.792534 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.792553 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.796079 (+ 3526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.796086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.798778 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.798805 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.802388 (+ 3583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.802395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.802609 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.802664 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 05:22:36.802863 (+ 199us) client_negotiation.cc:240] Negotiation successful
0809 05:22:36.803082 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":917,"threads_started":1}
I20260809 05:22:36.803542 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.788402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.22.1.67:33427)
0809 05:22:36.789167 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.789177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.789626 (+ 449us) server_negotiation.cc:407] Connection header received
0809 05:22:36.789953 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.789963 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.790244 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.790441 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:36.792793 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.794671 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.799022 (+ 4351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.801991 (+ 2969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.802123 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.802386 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.802943 (+ 557us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.803173 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":415,"threads_started":1}
I20260809 05:22:36.808127 6355 tablet_service.cc:1467] Processing CreateTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd (DEFAULT_TABLE table=test-table [id=58871b546a3f4b93af471ced89c5cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:36.809773 6355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e398e035e844cfb354f3a9c3fd6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:36.810838 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.790406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:35987 (local address 127.0.0.1:58032)
0809 05:22:36.802393 (+ 11987us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:36.802416 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:36.802554 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:36.803756 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:36.803760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:36.803786 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:36.804165 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.804178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.806356 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.806361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.808447 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.808463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.809963 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.809969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.810136 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.810168 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:22:36.810261 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 05:22:36.810393 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11830,"mutex_wait_us":1,"thread_start_us":11666,"threads_started":1}
I20260809 05:22:36.811614 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.790914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.22.1.66:35987)
0809 05:22:36.802827 (+ 11913us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.802837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.802868 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:22:36.802970 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.802979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.803274 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.803507 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:36.804726 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.806214 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.808627 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.809769 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.809889 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.810111 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.811130 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.811304 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11739,"thread_start_us":11633,"threads_started":1}
I20260809 05:22:36.812230 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.790198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.22.1.65:34491)
0809 05:22:36.804570 (+ 14372us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:36.804579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:36.804611 (+ 32us) server_negotiation.cc:407] Connection header received
0809 05:22:36.804729 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:36.804736 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:36.804984 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:36.805831 (+ 847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:36.806084 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.807149 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.809107 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.810745 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.810875 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.811637 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:36.811713 (+ 76us) server_negotiation.cc:299] Negotiation successful
0809 05:22:36.811870 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14178,"thread_start_us":14019,"threads_started":1}
I20260809 05:22:36.812273 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:36.789838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:34491 (local address 127.0.0.1:58986)
0809 05:22:36.790351 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:36.790368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:36.790486 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:36.805235 (+ 14749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:36.805240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:36.805267 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:36.805630 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.805642 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.807296 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.807299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.808927 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:36.808941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:36.811310 (+ 2369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:36.811316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:36.811483 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:36.811514 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 05:22:36.811605 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 05:22:36.811908 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":47,"thread_start_us":249,"threads_started":1}
I20260809 05:22:36.815506 5961 tablet_service.cc:1467] Processing CreateTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd (DEFAULT_TABLE table=test-table [id=58871b546a3f4b93af471ced89c5cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:36.815531 6153 tablet_service.cc:1467] Processing CreateTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd (DEFAULT_TABLE table=test-table [id=58871b546a3f4b93af471ced89c5cada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:22:36.816776 5961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e398e035e844cfb354f3a9c3fd6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:36.816956 6153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e398e035e844cfb354f3a9c3fd6edd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:22:36.824980 6493 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Bootstrap starting.
I20260809 05:22:36.829005 6494 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Bootstrap starting.
I20260809 05:22:36.830394 6493 tablet_bootstrap.cc:654] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:36.830958 6498 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Bootstrap starting.
I20260809 05:22:36.831904 6493 log.cc:824] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:36.832376 6494 tablet_bootstrap.cc:654] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:36.833565 6494 log.cc:824] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:36.834198 6498 tablet_bootstrap.cc:654] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Neither blocks nor log segments found. Creating new log.
I20260809 05:22:36.835264 6493 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: No bootstrap required, opened a new log
I20260809 05:22:36.835377 6498 log.cc:824] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Log is configured to *not* fsync() on all Append() calls
I20260809 05:22:36.835577 6493 ts_tablet_manager.cc:1397] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260809 05:22:36.836193 6494 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: No bootstrap required, opened a new log
I20260809 05:22:36.836467 6494 ts_tablet_manager.cc:1397] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 05:22:36.838308 6498 tablet_bootstrap.cc:492] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: No bootstrap required, opened a new log
I20260809 05:22:36.838608 6498 ts_tablet_manager.cc:1397] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 05:22:36.843705 6494 raft_consensus.cc:348] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.844071 6494 raft_consensus.cc:374] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:36.844182 6494 raft_consensus.cc:729] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fa703e88a17424f80f723ae952c5f32, State: Initialized, Role: FOLLOWER
I20260809 05:22:36.844791 6494 consensus_queue.cc:260] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.846659 6494 ts_tablet_manager.cc:1428] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260809 05:22:36.846452 6493 raft_consensus.cc:348] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.846876 6493 raft_consensus.cc:374] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:36.846958 6493 raft_consensus.cc:729] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7286056c3ce74410a50bebfbf4d26e32, State: Initialized, Role: FOLLOWER
I20260809 05:22:36.847651 6493 consensus_queue.cc:260] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.848805 6466 heartbeater.cc:503] Master 127.22.1.126:38411 was elected leader, sending a full tablet report...
I20260809 05:22:36.848610 6498 raft_consensus.cc:348] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.849048 6498 raft_consensus.cc:374] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:22:36.849143 6498 raft_consensus.cc:729] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55379cef0dd840c399c62010b3c01c47, State: Initialized, Role: FOLLOWER
I20260809 05:22:36.849520 6493 ts_tablet_manager.cc:1428] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260809 05:22:36.849903 6498 consensus_queue.cc:260] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:36.852542 6254 heartbeater.cc:503] Master 127.22.1.126:38411 was elected leader, sending a full tablet report...
I20260809 05:22:36.853344 6498 ts_tablet_manager.cc:1428] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260809 05:22:36.909547 6056 tablet.cc:2369] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:22:36.938905 6255 tablet.cc:2369] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:22:36.967682 6467 tablet.cc:2369] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:22:37.032784 6505 raft_consensus.cc:482] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:22:37.033107 6505 raft_consensus.cc:504] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:37.035056 6505 leader_election.cc:288] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7286056c3ce74410a50bebfbf4d26e32 (127.22.1.67:33427), 55379cef0dd840c399c62010b3c01c47 (127.22.1.66:35987)
I20260809 05:22:37.043597 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.035463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:58669 (local address 127.22.1.67:33427)
0809 05:22:37.035646 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.035651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.036302 (+ 651us) server_negotiation.cc:407] Connection header received
0809 05:22:37.036557 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.036562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.036652 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.036764 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:37.037618 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.038697 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.041222 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.042454 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.042587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.042708 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.043196 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.043333 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260809 05:22:37.043642 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.036638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:60575 (local address 127.22.1.66:35987)
0809 05:22:37.036908 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.036914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.037351 (+ 437us) server_negotiation.cc:407] Connection header received
0809 05:22:37.037516 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.037520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.037595 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.037713 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:22:37.038158 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.039059 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.041323 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.042524 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.042643 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.042726 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.043196 (+ 470us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.043333 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 05:22:37.043800 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.035458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33427 (local address 127.22.1.65:58669)
0809 05:22:37.036186 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:37.036211 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:37.036357 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:37.036957 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:37.036962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:37.036996 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:37.037457 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:37.037481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.038845 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.038849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:37.041031 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:37.041048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.042701 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.042708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:37.042993 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.043062 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 05:22:37.043157 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:22:37.043338 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":93,"thread_start_us":369,"threads_started":1}
I20260809 05:22:37.044204 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.036320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:35987 (local address 127.22.1.65:60575)
0809 05:22:37.037235 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0809 05:22:37.037254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:22:37.037390 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:22:37.037721 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:22:37.037724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:22:37.037744 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:22:37.038013 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:37.038021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.039185 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.039190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:37.041188 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:22:37.041200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.042736 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.042741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:22:37.042998 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.043065 (+ 67us) client_negotiation.cc:769] Sending connection context
0809 05:22:37.043157 (+ 92us) client_negotiation.cc:240] Negotiation successful
0809 05:22:37.043338 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":106,"thread_start_us":539,"threads_started":1}
I20260809 05:22:37.044886 6388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e398e035e844cfb354f3a9c3fd6edd" candidate_uuid: "7fa703e88a17424f80f723ae952c5f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7286056c3ce74410a50bebfbf4d26e32" is_pre_election: true
I20260809 05:22:37.045290 6183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e398e035e844cfb354f3a9c3fd6edd" candidate_uuid: "7fa703e88a17424f80f723ae952c5f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55379cef0dd840c399c62010b3c01c47" is_pre_election: true
I20260809 05:22:37.045590 6388 raft_consensus.cc:2454] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa703e88a17424f80f723ae952c5f32 in term 0.
I20260809 05:22:37.045835 6183 raft_consensus.cc:2454] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fa703e88a17424f80f723ae952c5f32 in term 0.
I20260809 05:22:37.046304 5884 leader_election.cc:302] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7286056c3ce74410a50bebfbf4d26e32, 7fa703e88a17424f80f723ae952c5f32; no voters:
I20260809 05:22:37.046700 6505 raft_consensus.cc:2781] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:22:37.046857 6505 raft_consensus.cc:482] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:22:37.046972 6505 raft_consensus.cc:3037] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:37.049211 6505 raft_consensus.cc:504] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:37.049817 6505 leader_election.cc:288] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [CANDIDATE]: Term 1 election: Requested vote from peers 7286056c3ce74410a50bebfbf4d26e32 (127.22.1.67:33427), 55379cef0dd840c399c62010b3c01c47 (127.22.1.66:35987)
I20260809 05:22:37.050267 6388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e398e035e844cfb354f3a9c3fd6edd" candidate_uuid: "7fa703e88a17424f80f723ae952c5f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7286056c3ce74410a50bebfbf4d26e32"
I20260809 05:22:37.050367 6183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e398e035e844cfb354f3a9c3fd6edd" candidate_uuid: "7fa703e88a17424f80f723ae952c5f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55379cef0dd840c399c62010b3c01c47"
I20260809 05:22:37.050503 6388 raft_consensus.cc:3037] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:37.050567 6183 raft_consensus.cc:3037] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:22:37.052690 6183 raft_consensus.cc:2454] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa703e88a17424f80f723ae952c5f32 in term 1.
I20260809 05:22:37.052686 6388 raft_consensus.cc:2454] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fa703e88a17424f80f723ae952c5f32 in term 1.
I20260809 05:22:37.053043 5884 leader_election.cc:302] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7286056c3ce74410a50bebfbf4d26e32, 7fa703e88a17424f80f723ae952c5f32; no voters:
I20260809 05:22:37.053326 6505 raft_consensus.cc:2781] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:22:37.053795 6505 raft_consensus.cc:686] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 1 LEADER]: Becoming Leader. State: Replica: 7fa703e88a17424f80f723ae952c5f32, State: Running, Role: LEADER
I20260809 05:22:37.054279 6505 consensus_queue.cc:237] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } }
I20260809 05:22:37.059262 5759 catalog_manager.cc:5495] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fa703e88a17424f80f723ae952c5f32 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "7fa703e88a17424f80f723ae952c5f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fa703e88a17424f80f723ae952c5f32" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 34491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 } health_report { overall_health: UNKNOWN } } }
I20260809 05:22:37.094964 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.089943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.22.1.65:34491)
0809 05:22:37.090137 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.090142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.090159 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:22:37.090307 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.090311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.090392 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.090558 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:22:37.090859 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.091611 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.092914 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.093403 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.093508 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.093951 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:22:37.094479 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:22:37.094603 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.094695 (+ 92us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.094765 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:22:37.103386 6388 raft_consensus.cc:1261] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 1 FOLLOWER]: Refusing update from remote peer 7fa703e88a17424f80f723ae952c5f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:37.103385 6183 raft_consensus.cc:1261] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 1 FOLLOWER]: Refusing update from remote peer 7fa703e88a17424f80f723ae952c5f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:22:37.104363 6516 consensus_queue.cc:1035] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55379cef0dd840c399c62010b3c01c47" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:37.104749 6505 consensus_queue.cc:1035] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7286056c3ce74410a50bebfbf4d26e32" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:22:37.115691 6519 mvcc.cc:205] Tried to move back new op lower bound from 7316492112285171712 to 7316492112097775616. Current Snapshot: MvccSnapshot[applied={T|T < 7316492112285171712}]
I20260809 05:22:37.121677 6520 mvcc.cc:205] Tried to move back new op lower bound from 7316492112285171712 to 7316492112097775616. Current Snapshot: MvccSnapshot[applied={T|T < 7316492112285171712}]
I20260809 05:22:37.130447 5759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:37.130704 5759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:22:37.134346 5759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b19fe625e404dbb827cab687ba7833e: Sending DeleteTablet for 3 replicas of tablet 18e398e035e844cfb354f3a9c3fd6edd
I20260809 05:22:37.135663 6355 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:37 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:56896
I20260809 05:22:37.135658 6153 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:37 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:58032
I20260809 05:22:37.135663 5961 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e398e035e844cfb354f3a9c3fd6edd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:22:37 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:58986
I20260809 05:22:37.138084 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 5858
I20260809 05:22:37.147879 6539 tablet_replica.cc:330] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: stopping tablet replica
I20260809 05:22:37.148571 6539 raft_consensus.cc:2229] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:22:37.149426 6539 raft_consensus.cc:2258] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:37.152303 6539 ts_tablet_manager.cc:1905] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:37.160799 6539 ts_tablet_manager.cc:1918] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:37.161175 6539 log.cc:1196] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/18e398e035e844cfb354f3a9c3fd6edd
I20260809 05:22:37.161878 6539 ts_tablet_manager.cc:1939] T 18e398e035e844cfb354f3a9c3fd6edd P 7fa703e88a17424f80f723ae952c5f32: Deleting consensus metadata
I20260809 05:22:37.164074 5733 catalog_manager.cc:4841] TS 7fa703e88a17424f80f723ae952c5f32 (127.22.1.65:34491): tablet 18e398e035e844cfb354f3a9c3fd6edd (table test-table [id=58871b546a3f4b93af471ced89c5cada]) successfully deleted
I20260809 05:22:37.165213 6537 tablet_replica.cc:330] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: stopping tablet replica
I20260809 05:22:37.165467 6538 tablet_replica.cc:330] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: stopping tablet replica
I20260809 05:22:37.165723 6537 raft_consensus.cc:2229] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:37.165891 6538 raft_consensus.cc:2229] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:22:37.166361 6537 raft_consensus.cc:2258] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:37.166361 6538 raft_consensus.cc:2258] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:22:37.168716 6538 ts_tablet_manager.cc:1905] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:22:37.169047 6537 ts_tablet_manager.cc:1905] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:22:37.175698 6538 ts_tablet_manager.cc:1918] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:37.175875 6538 log.cc:1196] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/18e398e035e844cfb354f3a9c3fd6edd
I20260809 05:22:37.176358 6537 ts_tablet_manager.cc:1918] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:22:37.176426 6538 ts_tablet_manager.cc:1939] T 18e398e035e844cfb354f3a9c3fd6edd P 55379cef0dd840c399c62010b3c01c47: Deleting consensus metadata
I20260809 05:22:37.176505 6537 log.cc:1196] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/18e398e035e844cfb354f3a9c3fd6edd
I20260809 05:22:37.176972 6537 ts_tablet_manager.cc:1939] T 18e398e035e844cfb354f3a9c3fd6edd P 7286056c3ce74410a50bebfbf4d26e32: Deleting consensus metadata
I20260809 05:22:37.178524 5733 catalog_manager.cc:4841] TS 7286056c3ce74410a50bebfbf4d26e32 (127.22.1.67:33427): tablet 18e398e035e844cfb354f3a9c3fd6edd (table test-table [id=58871b546a3f4b93af471ced89c5cada]) successfully deleted
I20260809 05:22:37.178838 5733 catalog_manager.cc:4841] TS 55379cef0dd840c399c62010b3c01c47 (127.22.1.66:35987): tablet 18e398e035e844cfb354f3a9c3fd6edd (table test-table [id=58871b546a3f4b93af471ced89c5cada]) successfully deleted
I20260809 05:22:37.182020 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.139577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.22.1.65:34491)
0809 05:22:37.139765 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.139772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.160423 (+ 20651us) server_negotiation.cc:407] Connection header received
0809 05:22:37.161040 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.161047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.161144 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.161251 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:37.163188 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.164542 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.167064 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.167787 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.167894 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.173526 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:37.173546 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:37.173567 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:37.173614 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:37.178010 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.179637 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.179645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.179648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.179813 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.180314 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.180317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.180319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.180766 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:37.181049 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.181442 (+ 393us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.181636 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:22:37.182731 6048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:37.287075 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 5858
I20260809 05:22:37.311697 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 6064
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:22:37.326561 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.312501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58044 (local address 127.22.1.66:35987)
0809 05:22:37.312717 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.312723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.312744 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:22:37.312914 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.312919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.313014 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.313115 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:37.314430 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.315202 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.316476 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.316906 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.317015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.321053 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:37.321069 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:37.321088 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:37.321130 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:37.323692 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.324370 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.324379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.324381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.324538 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.324941 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.324945 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.324948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.325533 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:37.325783 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.326096 (+ 313us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.326278 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260809 05:22:37.327270 6246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:37.416517 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 6064
I20260809 05:22:37.440128 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 6264
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[5685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:22:37.455196 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:22:37.440927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.22.1.67:33427)
0809 05:22:37.441134 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0809 05:22:37.441139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:22:37.441183 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:22:37.441298 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:22:37.441302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:22:37.441384 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:22:37.441460 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:22:37.442710 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.443437 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.444811 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:22:37.445255 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:22:37.445350 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:22:37.449684 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:22:37.449701 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:22:37.449720 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:22:37.449759 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:22:37.452349 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.453068 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.453076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.453078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.453280 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:22:37.453677 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:22:37.453680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:22:37.453682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:22:37.454074 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:22:37.454309 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0809 05:22:37.454636 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 05:22:37.454823 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 05:22:37.455910 6459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:37.543674 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 6264
I20260809 05:22:37.562469 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 5707
I20260809 05:22:37.563620 5809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:22:37.674352 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 5707
2026-08-09T05:22:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5552 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[6555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[6555](info): set up 2 sockets
Aug 09 05:22:37 dist-test-slave-rmhg krb5kdc[6555](info): commencing operation
krb5kdc: starting...
W20260809 05:22:38.879743 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:22:39 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:22:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangeradmin_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangeradmin_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangerlookup_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangerlookup_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
I20260809 05:22:39.269948 22533 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGKhvF4/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/postgres -l logfile start
2026-08-09 05:22:41.567 UTC [6603] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 05:22:41.567 UTC [6603] LOG: listening on IPv4 address "127.22.1.84", port 33437
2026-08-09 05:22:41.570 UTC [6603] LOG: listening on Unix socket "/tmp/.s.PGSQL.33437"
2026-08-09 05:22:41.592 UTC [6607] LOG: database system was shut down at 2026-08-09 05:22:40 UTC
2026-08-09 05:22:41.596 UTC [6603] LOG: database system is ready to accept connections
I20260809 05:22:42.674000 22533 mini_postgres.cc:96] Postgres bound to 33437
2026-08-09 05:22:42.695 UTC [6627] FATAL: database "slave" does not exist
127.22.1.84:33437 - accepting connections
I20260809 05:22:42.696678 22533 mini_ranger.cc:161] Starting Ranger...
I20260809 05:22:42.727464 22533 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 05:22:43.145254 22533 mini_ranger.cc:87] Created ranger Postgres database
I20260809 05:22:43.145426 22533 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin
2026-08-09 05:22:43,453 [I] DB FLAVOR :POSTGRES
2026-08-09 05:22:43,453 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 05:22:43,453 [I] Checking connection..
2026-08-09 05:22:43,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 05:22:43,720 [I] Checking connection passed.
2026-08-09 05:22:43,720 [I] --------- Verifying version history table ---------
2026-08-09 05:22:43,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:22:43,982 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 05:22:43,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:22:44,238 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 05:22:44,239 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 05:22:44,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 05:22:44.498 UTC [6732] WARNING: there is no transaction in progress
2026-08-09 05:22:44,506 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 05:22:44,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:22:44,770 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 05:22:44,771 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 05:22:44,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 05:22:45,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 05:22:45,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-09 05:22:45,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 05:22:45,762 [I] Table x_portal_user does not exist in database ranger
2026-08-09 05:22:45,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 05:22:46,004 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 05:22:46,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 05:22:46,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 05:22:46,528 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 05:22:46,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 05:22:47.487 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.502 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.518 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.533 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.548 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.569 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.627 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.634 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.648 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.658 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.668 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.682 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.691 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.697 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.703 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:47.709 UTC [6906] WARNING: there is no transaction in progress
2026-08-09 05:22:48,313 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 05:22:48,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 05:22:48,555 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 05:22:48,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 05:22:48,821 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 05:22:48,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 05:22:49,075 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 05:22:49,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 05:22:49,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 05:22:49,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-09 05:22:49,781 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 05:22:49,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 05:22:50,028 [I] DB_PATCHES have already been applied
I20260809 05:22:50.034427 22533 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGKhvF4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 05:22:50.034555 22533 mini_ranger.cc:193] Using host: 127.22.1.84
I20260809 05:22:50.055426 22533 mini_ranger.cc:227] Ranger admin URL: http://127.22.1.84:54965
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 5:22:50 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 05:22:51.260769 22533 mini_ranger.cc:160] Time spent starting Ranger: real 8.564s user 0.005s sys 0.055s
Aug 09, 2026 5:23:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 5:23:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Aug 09, 2026 5:23:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 5:23:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 5:23:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 05:23:11.881765 22533 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangerkms_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/rangerkms_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.1.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.1.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 05:23:12 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252992, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 05:23:12.101207 22533 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 05:23:12.128654 22533 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 05:23:12.914017 22533 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 05:23:12.914176 22533 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms
I20260809 05:23:12.914336 22533 mini_ranger_kms.cc:227] Using postgres at 127.22.1.84:33437
2026-08-09 05:23:13,393 [I] DB FLAVOR :POSTGRES
2026-08-09 05:23:13,394 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 05:23:13,394 [I] Checking connection
2026-08-09 05:23:13,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 05:23:13,654 [I] connection success
2026-08-09 05:23:13,654 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 05:23:13,654 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 05:23:13,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 05:23:13,891 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 05:23:13,891 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 05:23:13,891 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 05:23:13,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:33437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 05:23:14,144 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 05:23:14.591298 22533 mini_ranger_kms.cc:310] Created kms service
I20260809 05:23:14.795920 22533 mini_ranger_kms.cc:326] Created kudu user
I20260809 05:23:14.838900 22533 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 05:23:14 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252992, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:15.264189 22533 mini_ranger_kms.cc:384] Added ranger policy
I20260809 05:23:15.265774 22533 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 05:23:15.265861 22533 mini_ranger_kms.cc:240] Using host: 127.22.1.84
I20260809 05:23:15.294279 22533 mini_ranger_kms.cc:276] Ranger KMS PID: 7181
I20260809 05:23:15.294508 22533 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.1.84:45043
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 5:23:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 05:23:17 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786252997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:23:17.780249 22533 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.679s user 0.156s sys 0.368s
I20260809 05:23:17.780608 22533 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 05:23:17.780830 22533 mini_ranger_kms.cc:401] 127.22.1.84:45043/kms/v1/keys
Aug 09 05:23:18 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
Aug 09 05:23:18 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786252997, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
Aug 09 05:23:20 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:23:20.892349 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38145
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38145
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:45043/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:23:21.480432 7260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:23:21.480881 7260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:23:21.481067 7260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:23:21.491370 7260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:23:21.491544 7260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:23:21.491639 7260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:23:21.491731 7260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:23:21.491808 7260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:23:21.504698 7260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.1.84:45043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38145
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:23:21.506181 7260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:23:21.507664 7260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:23:21.524165 7271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:21.524394 7270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:21.524648 7277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:23:21.526170 7260 server_base.cc:1034] running on GCE node
I20260809 05:23:21.527101 7260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:23:21.528671 7260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:23:21.529964 7260 hybrid_clock.cc:648] HybridClock initialized: now 1786253001529835 us; error 32 us; skew 500 ppm
I20260809 05:23:21.530455 7260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:23:21 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:23:21.538504 7260 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:23:21.540831 7260 webserver.cc:460] Webserver started at http://127.22.1.126:46183/ using document root <none> and password file <none>
I20260809 05:23:21.541769 7260 fs_manager.cc:359] Metadata directory not provided
I20260809 05:23:21.541982 7260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:23:21.542397 7260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:23:21 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253001, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:21.601938 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "0f072fac8eef4dd98790031e975adb76"
format_stamp: "Formatted at 2026-08-09 05:23:21 on dist-test-slave-rmhg"
server_key: "c3347331c198c758ad4189ca9c979ec1"
server_key_iv: "a779c7d5ba644c3ff300549277b67259"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:21.603112 7260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "0f072fac8eef4dd98790031e975adb76"
format_stamp: "Formatted at 2026-08-09 05:23:21 on dist-test-slave-rmhg"
server_key: "c3347331c198c758ad4189ca9c979ec1"
server_key_iv: "a779c7d5ba644c3ff300549277b67259"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:21.609519 7260 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260809 05:23:21.635226 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:21.636937 7260 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260809 05:23:21.637228 7260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "0f072fac8eef4dd98790031e975adb76"
format_stamp: "Formatted at 2026-08-09 05:23:21 on dist-test-slave-rmhg"
server_key: "c3347331c198c758ad4189ca9c979ec1"
server_key_iv: "a779c7d5ba644c3ff300549277b67259"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:21.637547 7260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:23:21.652837 7260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:23:21.658429 7260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:23:21.659040 7260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:23:21.708580 7260 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38145
I20260809 05:23:21.708588 7359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38145 every 8 connection(s)
I20260809 05:23:21.711030 7260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:23:21.712003 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 7260
I20260809 05:23:21.712230 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:23:21.713069 22533 external_mini_cluster.cc:1428] Setting key e91e591bebb2ed72876ba3e0b6bdb4eb
I20260809 05:23:21.717353 7360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 09 05:23:21 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:23:21.737574 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: Bootstrap starting.
I20260809 05:23:21.742087 7360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: Neither blocks nor log segments found. Creating new log.
I20260809 05:23:21.744020 7360 log.cc:824] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: Log is configured to *not* fsync() on all Append() calls
I20260809 05:23:21.747413 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:21.715387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.22.1.126:38145)
0809 05:23:21.718466 (+ 3079us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:21.718485 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:21.718583 (+ 98us) server_negotiation.cc:407] Connection header received
0809 05:23:21.722584 (+ 4001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:21.722624 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:21.723292 (+ 668us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:21.723876 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:21.726710 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:21.728940 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:21.731073 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:21.732498 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:21.732724 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:21.737803 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:21.737819 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:21.737840 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:21.737890 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:21.741834 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:21.742728 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:21.742736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:21.742743 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:21.742997 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:21.743537 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:21.743542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:21.743544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:21.744329 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:21.744708 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:21.745657 (+ 949us) server_negotiation.cc:299] Negotiation successful
0809 05:23:21.746057 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2749,"thread_start_us":2373,"threads_started":1}
I20260809 05:23:21.747671 7360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: No bootstrap required, opened a new log
I20260809 05:23:21.758412 7360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } }
I20260809 05:23:21.758910 7360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:23:21.759016 7360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f072fac8eef4dd98790031e975adb76, State: Initialized, Role: FOLLOWER
I20260809 05:23:21.759699 7360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } }
I20260809 05:23:21.759917 7360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:23:21.760044 7360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:23:21.760201 7360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:23:21.762413 7360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } }
I20260809 05:23:21.762934 7360 leader_election.cc:302] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f072fac8eef4dd98790031e975adb76; no voters:
I20260809 05:23:21.763639 7360 leader_election.cc:288] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:23:21.763929 7371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:23:21.764832 7371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [term 1 LEADER]: Becoming Leader. State: Replica: 0f072fac8eef4dd98790031e975adb76, State: Running, Role: LEADER
I20260809 05:23:21.765421 7371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } }
I20260809 05:23:21.766026 7360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:23:21.768885 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f072fac8eef4dd98790031e975adb76. Latest consensus state: current_term: 1 leader_uuid: "0f072fac8eef4dd98790031e975adb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } } }
I20260809 05:23:21.769802 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [sys.catalog]: This master's current role is: LEADER
I20260809 05:23:21.769057 7372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f072fac8eef4dd98790031e975adb76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f072fac8eef4dd98790031e975adb76" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38145 } } }
I20260809 05:23:21.770179 7372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76 [sys.catalog]: This master's current role is: LEADER
I20260809 05:23:21.778890 7384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:23:21.789032 7384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:23:21.790138 7360 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 05:23:21.811967 7384 catalog_manager.cc:1347] Generated new cluster ID: 797493188c6c4de7bf61677b14d9e657
I20260809 05:23:21.812119 7384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:23:21.831519 7384 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:23:21.834053 7384 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:23:21.848230 7384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: Generated new TSK 0
I20260809 05:23:21.849625 7384 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:23:22.687052 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:45043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:23:22 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786253002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 05:23:23.271863 7435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:23:23.272353 7435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:23:23.272568 7435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:23:23.283207 7435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:23:23.283420 7435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:23:23.283629 7435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:23:23.296463 7435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.1.84:45043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:23:23.298337 7435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:23:23.299960 7435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:23:23.317770 7454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:23.318245 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:23.318765 7453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:23:23.320251 7435 server_base.cc:1034] running on GCE node
I20260809 05:23:23.321280 7435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:23:23.322805 7435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:23:23.324179 7435 hybrid_clock.cc:648] HybridClock initialized: now 1786253003323976 us; error 91 us; skew 500 ppm
I20260809 05:23:23.324651 7435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:23:23 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:23:23.330288 7435 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:23:23.354070 7435 webserver.cc:460] Webserver started at http://127.22.1.65:35761/ using document root <none> and password file <none>
I20260809 05:23:23.355172 7435 fs_manager.cc:359] Metadata directory not provided
I20260809 05:23:23.355365 7435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:23:23.355803 7435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:23:23 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:23.379670 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "ece7026890434da59563ef0fd2e0d88a"
format_stamp: "Formatted at 2026-08-09 05:23:23 on dist-test-slave-rmhg"
server_key: "30ae176debfd557a524903229912173c"
server_key_iv: "ee625a7975aedb2e8e56f5172abad339"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:23.381549 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "ece7026890434da59563ef0fd2e0d88a"
format_stamp: "Formatted at 2026-08-09 05:23:23 on dist-test-slave-rmhg"
server_key: "30ae176debfd557a524903229912173c"
server_key_iv: "ee625a7975aedb2e8e56f5172abad339"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:23.394310 7435 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.008s
I20260809 05:23:23.434259 7468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:23.436573 7435 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.011s
I20260809 05:23:23.436807 7435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "ece7026890434da59563ef0fd2e0d88a"
format_stamp: "Formatted at 2026-08-09 05:23:23 on dist-test-slave-rmhg"
server_key: "30ae176debfd557a524903229912173c"
server_key_iv: "ee625a7975aedb2e8e56f5172abad339"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:23.437134 7435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:23:23.470042 7435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:23:23.477030 7435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:23:23.477442 7435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:23:23.479369 7435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:23:23.482499 7435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:23:23.482692 7435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:23.482864 7435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:23:23.482936 7435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:23:23 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:23.658068 7435 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:40573
I20260809 05:23:23.658080 7656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:40573 every 8 connection(s)
I20260809 05:23:23.660427 7435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:23:23.669529 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 7435
I20260809 05:23:23.669762 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:23:23.670759 22533 external_mini_cluster.cc:1428] Setting key 1a843d47c1d77f5078632908b3383d16
Aug 09 05:23:23 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253003, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:23:23.691691 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:23.663975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:51041 (local address 127.22.1.126:38145)
0809 05:23:23.664571 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:23.664578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:23.666011 (+ 1433us) server_negotiation.cc:407] Connection header received
0809 05:23:23.668187 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:23.668193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:23.668359 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:23.668505 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:23.670680 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:23.672244 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:23.675031 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:23.676080 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:23.676225 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:23.682067 (+ 5842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:23.682074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:23.682079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:23.682120 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:23.686938 (+ 4818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:23.688045 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:23.688051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:23.688054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:23.688373 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:23.689026 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:23.689031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:23.689034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:23.689530 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:23.689674 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:23.690747 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0809 05:23:23.690983 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":315,"threads_started":1}
I20260809 05:23:23.695333 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:23.664698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38145 (local address 127.22.1.65:51041)
0809 05:23:23.665806 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:23.665865 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:23.667885 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:23.668690 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:23.668699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:23.669488 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:23:23.670445 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:23.670477 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:23.672492 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:23.672498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:23.674839 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:23.674857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:23.676472 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:23.676482 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:23.676865 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:23.677966 (+ 1101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:23:23.677997 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:23:23.681732 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:23:23.687132 (+ 5400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:23.687144 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:23.687167 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:23.687879 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:23.688514 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:23.688520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:23.688524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:23.688876 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:23.689678 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:23:23.689688 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:23:23.690189 (+ 501us) client_negotiation.cc:769] Sending connection context
0809 05:23:23.690564 (+ 375us) client_negotiation.cc:240] Negotiation successful
0809 05:23:23.691181 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":586,"threads_started":1}
I20260809 05:23:23.697741 7658 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38145
I20260809 05:23:23.698107 7658 heartbeater.cc:464] Registering TS with master...
I20260809 05:23:23.698881 7658 heartbeater.cc:511] Master 127.22.1.126:38145 requested a full tablet report, sending...
I20260809 05:23:23.701824 7310 ts_manager.cc:194] Registered new tserver with Master: ece7026890434da59563ef0fd2e0d88a (127.22.1.65:40573)
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
I20260809 05:23:23.704381 7310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:51041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:23:23.742519 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:45043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:23:23 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253002, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
W20260809 05:23:24.368135 7666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:23:24.368434 7666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:23:24.368567 7666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:23:24.376474 7666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:23:24.376735 7666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:23:24.376933 7666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:23:24.389325 7666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.1.84:45043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:23:24.390755 7666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:23:24.392133 7666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:23:24.405813 7688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:24.406612 7687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:24.406836 7696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:23:24.406870 7666 server_base.cc:1034] running on GCE node
I20260809 05:23:24.407593 7666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:23:24.408561 7666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:23:24.409767 7666 hybrid_clock.cc:648] HybridClock initialized: now 1786253004409707 us; error 50 us; skew 500 ppm
I20260809 05:23:24.410096 7666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:23:24 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:23:24.413477 7666 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:23:24.416464 7666 webserver.cc:460] Webserver started at http://127.22.1.66:34263/ using document root <none> and password file <none>
I20260809 05:23:24.417172 7666 fs_manager.cc:359] Metadata directory not provided
I20260809 05:23:24.417308 7666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:23:24.417706 7666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:23:24 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:24.436177 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "04812b5f90ff405b916c480bcae54260"
format_stamp: "Formatted at 2026-08-09 05:23:24 on dist-test-slave-rmhg"
server_key: "31c120391665098dec561139e56b0a0e"
server_key_iv: "9f690128a48fce0d87dc5023d6805df4"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:24.437173 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "04812b5f90ff405b916c480bcae54260"
format_stamp: "Formatted at 2026-08-09 05:23:24 on dist-test-slave-rmhg"
server_key: "31c120391665098dec561139e56b0a0e"
server_key_iv: "9f690128a48fce0d87dc5023d6805df4"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:24.444037 7666 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260809 05:23:24.461331 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:24.463165 7666 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260809 05:23:24.463353 7666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "04812b5f90ff405b916c480bcae54260"
format_stamp: "Formatted at 2026-08-09 05:23:24 on dist-test-slave-rmhg"
server_key: "31c120391665098dec561139e56b0a0e"
server_key_iv: "9f690128a48fce0d87dc5023d6805df4"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:24.463604 7666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:23:24.486482 7666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:23:24.491888 7666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:23:24.492336 7666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:23:24.493961 7666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:23:24.495682 7666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:23:24.495770 7666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:24.495916 7666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:23:24.496062 7666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:24.557832 7666 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:32885
I20260809 05:23:24.557971 7882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:32885 every 8 connection(s)
I20260809 05:23:24.559388 7666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:23:24.568326 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 7666
I20260809 05:23:24.568495 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:23:24.568940 22533 external_mini_cluster.cc:1428] Setting key 1beb0a133c4f23a7c67c3b13cf412024
Aug 09 05:23:24 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253004, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:23:24.586071 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:24.562793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:45745 (local address 127.22.1.126:38145)
0809 05:23:24.563451 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:24.563458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:24.564708 (+ 1250us) server_negotiation.cc:407] Connection header received
0809 05:23:24.567359 (+ 2651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:24.567370 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:24.567492 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:24.567598 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:24.569836 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:24.570861 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:24.572737 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:24.573446 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:24.573588 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:24.578077 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:24.578084 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:24.578088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:24.578155 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:24.582089 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:24.583253 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:24.583259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:24.583262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:24.583441 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:24.583987 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:24.583993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:24.583996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:24.584394 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:24.584549 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:24.585385 (+ 836us) server_negotiation.cc:299] Negotiation successful
0809 05:23:24.585630 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":412,"threads_started":1}
I20260809 05:23:24.587251 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:24.563494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38145 (local address 127.22.1.66:45745)
0809 05:23:24.564493 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:24.564571 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:24.567014 (+ 2443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:24.567958 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:24.567972 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:24.568668 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:23:24.569638 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:24.569670 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:24.571097 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:24.571103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:24.572576 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:24.572590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:24.573731 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:24.573738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:24.574016 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:24.574871 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:23:24.574905 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:23:24.577807 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:23:24.582279 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:24.582292 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:24.582315 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:24.583094 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:24.583537 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:24.583540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:24.583542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:24.583856 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:24.584524 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:23:24.584531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:23:24.584918 (+ 387us) client_negotiation.cc:769] Sending connection context
0809 05:23:24.585237 (+ 319us) client_negotiation.cc:240] Negotiation successful
0809 05:23:24.585779 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":419,"threads_started":1}
I20260809 05:23:24.588893 7886 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38145
I20260809 05:23:24.589274 7886 heartbeater.cc:464] Registering TS with master...
I20260809 05:23:24.590349 7886 heartbeater.cc:511] Master 127.22.1.126:38145 requested a full tablet report, sending...
I20260809 05:23:24.592154 7310 ts_manager.cc:194] Registered new tserver with Master: 04812b5f90ff405b916c480bcae54260 (127.22.1.66:32885)
I20260809 05:23:24.593288 7310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:45745
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:23:24.633514 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:45043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:23:24.708765 7658 heartbeater.cc:503] Master 127.22.1.126:38145 was elected leader, sending a full tablet report...
W20260809 05:23:25.185357 7896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:23:25.185755 7896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:23:25.185881 7896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:23:25.195662 7896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:23:25.195854 7896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:23:25.196041 7896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:23:25.210328 7896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.1.84:45043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38145
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:23:25.211959 7896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:23:25.213593 7896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:23:25.232345 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:25.232362 7903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:23:25.232600 7902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:23:25.233973 7896 server_base.cc:1034] running on GCE node
I20260809 05:23:25.234961 7896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:23:25.236116 7896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:23:25.237385 7896 hybrid_clock.cc:648] HybridClock initialized: now 1786253005237241 us; error 70 us; skew 500 ppm
I20260809 05:23:25.237859 7896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:23:25 dist-test-slave-rmhg krb5kdc[6555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253005, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:23:25.242008 7896 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:23:25.244029 7896 webserver.cc:460] Webserver started at http://127.22.1.67:33687/ using document root <none> and password file <none>
I20260809 05:23:25.244881 7896 fs_manager.cc:359] Metadata directory not provided
I20260809 05:23:25.245107 7896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:23:25.245609 7896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:23:25 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253005, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:23:25.263371 7896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "0ee292416b2d477ca58db81628c01bed"
format_stamp: "Formatted at 2026-08-09 05:23:25 on dist-test-slave-rmhg"
server_key: "fc8b206d34c188518a2c5275205de616"
server_key_iv: "8912da56b21ddc97981d88a9ba7ab062"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:25.264500 7896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "0ee292416b2d477ca58db81628c01bed"
format_stamp: "Formatted at 2026-08-09 05:23:25 on dist-test-slave-rmhg"
server_key: "fc8b206d34c188518a2c5275205de616"
server_key_iv: "8912da56b21ddc97981d88a9ba7ab062"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:25.271463 7896 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.007s
I20260809 05:23:25.290486 7915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:25.292333 7896 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 05:23:25.292550 7896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "0ee292416b2d477ca58db81628c01bed"
format_stamp: "Formatted at 2026-08-09 05:23:25 on dist-test-slave-rmhg"
server_key: "fc8b206d34c188518a2c5275205de616"
server_key_iv: "8912da56b21ddc97981d88a9ba7ab062"
server_key_version: "kuduclusterkey@0"
I20260809 05:23:25.292838 7896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:23:25.320109 7896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:23:25.325587 7896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:23:25.326153 7896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:23:25.327872 7896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:23:25.330112 7896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:23:25.330263 7896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:25.330485 7896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:23:25.330629 7896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:23:25.405192 7896 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:44325
I20260809 05:23:25.405296 8089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:44325 every 8 connection(s)
I20260809 05:23:25.406733 7896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:23:25.417202 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 7896
I20260809 05:23:25.417407 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:23:25.417941 22533 external_mini_cluster.cc:1428] Setting key d6a10a471eeba27ba006785f0a77cc3c
Aug 09 05:23:25 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253005, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:23:25.431555 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.409797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:59945 (local address 127.22.1.126:38145)
0809 05:23:25.410602 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.410608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.411578 (+ 970us) server_negotiation.cc:407] Connection header received
0809 05:23:25.413998 (+ 2420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.414005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.414186 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.414295 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:25.416118 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.417031 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.419288 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.420158 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.420295 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.424986 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:25.424991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:25.424995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:25.425031 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:25.428240 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.429028 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.429032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.429034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.429201 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.429687 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.429691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.429693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.430025 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:25.430152 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.430922 (+ 770us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.431151 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":486,"threads_started":1}
I20260809 05:23:25.432716 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.410440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38145 (local address 127.22.1.67:59945)
0809 05:23:25.411390 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.411434 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.413700 (+ 2266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.414596 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.414605 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.415190 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:23:25.415925 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.415947 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.417255 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.417259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.419082 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.419099 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.420548 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.420560 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.420949 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.421957 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:23:25.421997 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:23:25.424726 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:23:25.428380 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:25.428387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:25.428402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:25.428924 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:25.429301 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:23:25.429305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:23:25.429307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:23:25.429569 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:23:25.430149 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:23:25.430156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:23:25.430560 (+ 404us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.430816 (+ 256us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.431262 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":414,"threads_started":1}
I20260809 05:23:25.434578 8091 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38145
I20260809 05:23:25.434898 8091 heartbeater.cc:464] Registering TS with master...
I20260809 05:23:25.435676 8091 heartbeater.cc:511] Master 127.22.1.126:38145 requested a full tablet report, sending...
I20260809 05:23:25.437204 7310 ts_manager.cc:194] Registered new tserver with Master: 0ee292416b2d477ca58db81628c01bed (127.22.1.67:44325)
I20260809 05:23:25.438226 7310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:59945
I20260809 05:23:25.446928 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:23:25.469225 8095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.452597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.22.1.126:38145)
0809 05:23:25.452815 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.452822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.453559 (+ 737us) server_negotiation.cc:407] Connection header received
0809 05:23:25.453779 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.453783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.453866 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.453961 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:25.455813 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.456820 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.458552 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.459143 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.459256 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.462685 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:25.462691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:25.462695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:25.462736 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:25.466006 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.466906 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.466910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.466913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.467066 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.467556 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.467585 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.467587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.467969 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:25.468239 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.468589 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.468880 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:23:25.474035 7310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:23:25.481761 7310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:23:25.524766 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.499187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.22.1.67:44325)
0809 05:23:25.510947 (+ 11760us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.510957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.513566 (+ 2609us) server_negotiation.cc:407] Connection header received
0809 05:23:25.513718 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.513726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.513996 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.514251 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:23:25.515700 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.517618 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.521798 (+ 4180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.523298 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.523444 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.523806 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.524183 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.524364 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11426,"thread_start_us":10934,"threads_started":1}
I20260809 05:23:25.525720 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.495928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45296)
0809 05:23:25.498226 (+ 2298us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.498286 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.498495 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.513438 (+ 14943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.513447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.513494 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:23:25.514354 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.514372 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.517340 (+ 2968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.517346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.521790 (+ 4444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.521803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.524593 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.524603 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.524824 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.524860 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.524957 (+ 97us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.525105 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2019,"mutex_wait_us":30,"thread_start_us":15754,"threads_started":1}
I20260809 05:23:25.525790 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.496104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45296 (local address 127.22.1.65:40573)
0809 05:23:25.511836 (+ 15732us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.511844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.511880 (+ 36us) server_negotiation.cc:407] Connection header received
0809 05:23:25.511995 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.512004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.512362 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.512700 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:23:25.514641 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.517113 (+ 2472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.522021 (+ 4908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.523757 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.523888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.524188 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.525238 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.525420 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15457,"thread_start_us":15243,"threads_started":1}
I20260809 05:23:25.527043 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.496787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44325 (local address 127.0.0.1:43554)
0809 05:23:25.513230 (+ 16443us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.513254 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.513409 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.514739 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.514744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.514773 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:23:25.515469 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.515486 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.517825 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.517832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.521602 (+ 3770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.521618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.523601 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.523610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.523857 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.523916 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.524061 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.524290 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15087,"mutex_wait_us":1,"thread_start_us":953,"threads_started":1}
I20260809 05:23:25.530501 7551 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 (DEFAULT_TABLE table=test-table [id=8b75364a12254a4ca27caeea87613906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:23:25.531225 7995 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 (DEFAULT_TABLE table=test-table [id=8b75364a12254a4ca27caeea87613906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:23:25.532452 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.512214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:32885 (local address 127.0.0.1:47822)
0809 05:23:25.520580 (+ 8366us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.520601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.520789 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.521659 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.521665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.521694 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:23:25.522719 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.522734 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.524808 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.524813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.528731 (+ 3918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.528749 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.531229 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.531239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.531476 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.531511 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.531627 (+ 116us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.531775 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8181,"thread_start_us":7389,"threads_started":1}
I20260809 05:23:25.532648 7551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f7d20198944bc9b8161f6315065c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:23:25.532474 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.512898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.22.1.66:32885)
0809 05:23:25.518571 (+ 5673us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.518580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.520731 (+ 2151us) server_negotiation.cc:407] Connection header received
0809 05:23:25.520993 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.521003 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.521392 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.521665 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:23:25.522978 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.524617 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.528951 (+ 4334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.530856 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.531007 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.531254 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.531698 (+ 444us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.531912 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5457,"thread_start_us":5270,"threads_started":1}
I20260809 05:23:25.533233 7995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f7d20198944bc9b8161f6315065c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:23:25.537340 7792 tablet_service.cc:1467] Processing CreateTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 (DEFAULT_TABLE table=test-table [id=8b75364a12254a4ca27caeea87613906]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:23:25.539304 7792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e7f7d20198944bc9b8161f6315065c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:23:25.550683 8122 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Bootstrap starting.
I20260809 05:23:25.554309 8123 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Bootstrap starting.
I20260809 05:23:25.556553 8122 tablet_bootstrap.cc:654] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Neither blocks nor log segments found. Creating new log.
I20260809 05:23:25.559026 8122 log.cc:824] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Log is configured to *not* fsync() on all Append() calls
I20260809 05:23:25.560258 8123 tablet_bootstrap.cc:654] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Neither blocks nor log segments found. Creating new log.
I20260809 05:23:25.562748 8123 log.cc:824] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Log is configured to *not* fsync() on all Append() calls
I20260809 05:23:25.577314 8124 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Bootstrap starting.
I20260809 05:23:25.577744 8122 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: No bootstrap required, opened a new log
I20260809 05:23:25.578284 8122 ts_tablet_manager.cc:1397] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.011s
I20260809 05:23:25.579269 8123 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: No bootstrap required, opened a new log
I20260809 05:23:25.579739 8123 ts_tablet_manager.cc:1397] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.013s
I20260809 05:23:25.582795 8124 tablet_bootstrap.cc:654] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Neither blocks nor log segments found. Creating new log.
I20260809 05:23:25.584551 8124 log.cc:824] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Log is configured to *not* fsync() on all Append() calls
I20260809 05:23:25.588426 8124 tablet_bootstrap.cc:492] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: No bootstrap required, opened a new log
I20260809 05:23:25.588825 8124 ts_tablet_manager.cc:1397] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260809 05:23:25.593183 8122 raft_consensus.cc:348] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.593904 8122 raft_consensus.cc:374] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:23:25.593758 8123 raft_consensus.cc:348] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.594120 8122 raft_consensus.cc:729] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee292416b2d477ca58db81628c01bed, State: Initialized, Role: FOLLOWER
I20260809 05:23:25.594323 8123 raft_consensus.cc:374] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:23:25.594480 8123 raft_consensus.cc:729] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece7026890434da59563ef0fd2e0d88a, State: Initialized, Role: FOLLOWER
I20260809 05:23:25.595482 8123 consensus_queue.cc:260] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.595654 8122 consensus_queue.cc:260] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.596635 8133 raft_consensus.cc:482] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:23:25.597705 7886 heartbeater.cc:503] Master 127.22.1.126:38145 was elected leader, sending a full tablet report...
I20260809 05:23:25.599341 8123 ts_tablet_manager.cc:1428] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260809 05:23:25.596992 8133 raft_consensus.cc:504] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.602784 8091 heartbeater.cc:503] Master 127.22.1.126:38145 was elected leader, sending a full tablet report...
I20260809 05:23:25.604218 8133 leader_election.cc:288] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ece7026890434da59563ef0fd2e0d88a (127.22.1.65:40573), 04812b5f90ff405b916c480bcae54260 (127.22.1.66:32885)
I20260809 05:23:25.607470 8122 ts_tablet_manager.cc:1428] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Time spent starting tablet: real 0.029s user 0.015s sys 0.005s
I20260809 05:23:25.609405 8124 raft_consensus.cc:348] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.610169 8124 raft_consensus.cc:374] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:23:25.610359 8124 raft_consensus.cc:729] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04812b5f90ff405b916c480bcae54260, State: Initialized, Role: FOLLOWER
I20260809 05:23:25.611231 8124 consensus_queue.cc:260] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.614969 8124 ts_tablet_manager.cc:1428] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Time spent starting tablet: real 0.026s user 0.006s sys 0.012s
I20260809 05:23:25.615406 8096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.605521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.22.1.67:52879)
0809 05:23:25.605722 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.605744 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.605866 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.607706 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.607711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.607751 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:23:25.608105 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.608117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.610074 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.610079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.612288 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.612309 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.614435 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.614442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.614731 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.614769 (+ 38us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.614903 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.615052 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":1920}
I20260809 05:23:25.615924 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.606818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:52879 (local address 127.22.1.65:40573)
0809 05:23:25.607087 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.607094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.607113 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:23:25.607182 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.607187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.607264 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.607365 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:23:25.608265 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.609908 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.612475 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.614032 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.614215 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.614337 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.615448 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.615626 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:23:25.617688 7584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f7d20198944bc9b8161f6315065c8" candidate_uuid: "0ee292416b2d477ca58db81628c01bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece7026890434da59563ef0fd2e0d88a" is_pre_election: true
I20260809 05:23:25.618806 7584 raft_consensus.cc:2454] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee292416b2d477ca58db81628c01bed in term 0.
I20260809 05:23:25.619967 7919 leader_election.cc:302] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee292416b2d477ca58db81628c01bed, ece7026890434da59563ef0fd2e0d88a; no voters:
I20260809 05:23:25.620473 8136 raft_consensus.cc:2781] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:23:25.620731 8136 raft_consensus.cc:482] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:23:25.621026 8136 raft_consensus.cc:3037] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:23:25.622668 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.606639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:60263 (local address 127.22.1.66:32885)
0809 05:23:25.606936 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.606943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.612790 (+ 5847us) server_negotiation.cc:407] Connection header received
0809 05:23:25.612944 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.612959 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.613462 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.613563 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:23:25.615014 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.616246 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.619966 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.621437 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.621591 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.621667 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.622239 (+ 572us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.622351 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:23:25.623435 8137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.606032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:32885 (local address 127.22.1.67:60263)
0809 05:23:25.612680 (+ 6648us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.612698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:23:25.612837 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:23:25.613779 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:23:25.613783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:23:25.613810 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:23:25.614733 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.614750 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.616500 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.616505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.619742 (+ 3237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:23:25.619758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.621786 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.621796 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:23:25.622046 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.622086 (+ 40us) client_negotiation.cc:769] Sending connection context
0809 05:23:25.622188 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 05:23:25.622332 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6508,"thread_start_us":6260,"threads_started":1}
I20260809 05:23:25.624887 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f7d20198944bc9b8161f6315065c8" candidate_uuid: "0ee292416b2d477ca58db81628c01bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04812b5f90ff405b916c480bcae54260" is_pre_election: true
I20260809 05:23:25.625099 8136 raft_consensus.cc:504] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.625661 7824 raft_consensus.cc:2454] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee292416b2d477ca58db81628c01bed in term 0.
I20260809 05:23:25.626302 8136 leader_election.cc:288] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [CANDIDATE]: Term 1 election: Requested vote from peers ece7026890434da59563ef0fd2e0d88a (127.22.1.65:40573), 04812b5f90ff405b916c480bcae54260 (127.22.1.66:32885)
I20260809 05:23:25.626703 7584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f7d20198944bc9b8161f6315065c8" candidate_uuid: "0ee292416b2d477ca58db81628c01bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ece7026890434da59563ef0fd2e0d88a"
I20260809 05:23:25.626921 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e7f7d20198944bc9b8161f6315065c8" candidate_uuid: "0ee292416b2d477ca58db81628c01bed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04812b5f90ff405b916c480bcae54260"
I20260809 05:23:25.627029 7584 raft_consensus.cc:3037] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:23:25.627218 7824 raft_consensus.cc:3037] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:23:25.631471 7824 raft_consensus.cc:2454] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee292416b2d477ca58db81628c01bed in term 1.
I20260809 05:23:25.631778 7584 raft_consensus.cc:2454] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee292416b2d477ca58db81628c01bed in term 1.
I20260809 05:23:25.631991 7919 leader_election.cc:302] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04812b5f90ff405b916c480bcae54260, 0ee292416b2d477ca58db81628c01bed; no voters:
I20260809 05:23:25.632325 8136 raft_consensus.cc:2781] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:23:25.632557 8136 raft_consensus.cc:686] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 1 LEADER]: Becoming Leader. State: Replica: 0ee292416b2d477ca58db81628c01bed, State: Running, Role: LEADER
I20260809 05:23:25.633136 8136 consensus_queue.cc:237] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } }
I20260809 05:23:25.639117 7308 catalog_manager.cc:5495] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee292416b2d477ca58db81628c01bed (127.22.1.67). New cstate: current_term: 1 leader_uuid: "0ee292416b2d477ca58db81628c01bed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee292416b2d477ca58db81628c01bed" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 } health_report { overall_health: UNKNOWN } } }
W20260809 05:23:25.659093 8094 tablet.cc:2369] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:23:25.665006 7660 tablet.cc:2369] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:23:25.671667 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.664206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.22.1.67:44325)
0809 05:23:25.664393 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.664398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.664414 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:23:25.664514 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.664518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.664649 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.664815 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:23:25.665475 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.666507 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.669254 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.669797 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.669900 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.670555 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:23:25.671094 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:23:25.671219 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.671322 (+ 103us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.671398 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:23:25.682474 7584 raft_consensus.cc:1261] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 1 FOLLOWER]: Refusing update from remote peer 0ee292416b2d477ca58db81628c01bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:23:25.682489 7824 raft_consensus.cc:1261] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 1 FOLLOWER]: Refusing update from remote peer 0ee292416b2d477ca58db81628c01bed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:23:25.683488 8133 consensus_queue.cc:1035] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "04812b5f90ff405b916c480bcae54260" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 32885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:23:25.683887 8136 consensus_queue.cc:1035] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ece7026890434da59563ef0fd2e0d88a" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:23:25.717088 7310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:23:25.717613 7310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:23:25.723170 7310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f072fac8eef4dd98790031e975adb76: Sending DeleteTablet for 3 replicas of tablet 7e7f7d20198944bc9b8161f6315065c8
I20260809 05:23:25.725307 7551 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:23:25 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:45296
I20260809 05:23:25.725629 7995 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:23:25 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:43554
I20260809 05:23:25.727905 7792 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e7f7d20198944bc9b8161f6315065c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:23:25 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:47822
I20260809 05:23:25.730114 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 7435
I20260809 05:23:25.738629 8162 tablet_replica.cc:330] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: stopping tablet replica
I20260809 05:23:25.739365 8162 raft_consensus.cc:2229] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:23:25.740350 8162 raft_consensus.cc:2258] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:23:25.741262 8161 tablet_replica.cc:330] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: stopping tablet replica
I20260809 05:23:25.741814 8161 raft_consensus.cc:2229] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:23:25.742730 8161 raft_consensus.cc:2258] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:23:25.743593 8162 ts_tablet_manager.cc:1905] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:23:25.748134 8160 tablet_replica.cc:330] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: stopping tablet replica
I20260809 05:23:25.748703 8160 raft_consensus.cc:2229] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:23:25.749492 8160 raft_consensus.cc:2258] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 09 05:23:25 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:23:25.751374 8161 ts_tablet_manager.cc:1905] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:23:25.751967 8160 ts_tablet_manager.cc:1905] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:23:25.753724 8162 ts_tablet_manager.cc:1918] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:23:25.753940 8162 log.cc:1196] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/7e7f7d20198944bc9b8161f6315065c8
I20260809 05:23:25.754556 8162 ts_tablet_manager.cc:1939] T 7e7f7d20198944bc9b8161f6315065c8 P 04812b5f90ff405b916c480bcae54260: Deleting consensus metadata
I20260809 05:23:25.756798 7292 catalog_manager.cc:4841] TS 04812b5f90ff405b916c480bcae54260 (127.22.1.66:32885): tablet 7e7f7d20198944bc9b8161f6315065c8 (table test-table [id=8b75364a12254a4ca27caeea87613906]) successfully deleted
I20260809 05:23:25.761485 8161 ts_tablet_manager.cc:1918] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:23:25.761724 8161 log.cc:1196] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/7e7f7d20198944bc9b8161f6315065c8
I20260809 05:23:25.762240 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.732230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.22.1.65:40573)
0809 05:23:25.732425 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.732432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.737197 (+ 4765us) server_negotiation.cc:407] Connection header received
0809 05:23:25.737440 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.737447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.737556 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.737661 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:25.739945 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.741004 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.743415 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.744770 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.744887 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.752178 (+ 7291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:25.752195 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:25.752214 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:25.752298 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:25.757282 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.758417 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.758440 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.758443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.758740 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.759312 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.759317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.759321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.760015 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:25.760474 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.761189 (+ 715us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.761460 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:23:25.762614 8161 ts_tablet_manager.cc:1939] T 7e7f7d20198944bc9b8161f6315065c8 P 0ee292416b2d477ca58db81628c01bed: Deleting consensus metadata
I20260809 05:23:25.763547 8160 ts_tablet_manager.cc:1918] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:23:25.763798 8160 log.cc:1196] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/7e7f7d20198944bc9b8161f6315065c8
I20260809 05:23:25.764079 7649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:23:25.765009 7287 catalog_manager.cc:4841] TS 0ee292416b2d477ca58db81628c01bed (127.22.1.67:44325): tablet 7e7f7d20198944bc9b8161f6315065c8 (table test-table [id=8b75364a12254a4ca27caeea87613906]) successfully deleted
I20260809 05:23:25.915861 8160 ts_tablet_manager.cc:1939] T 7e7f7d20198944bc9b8161f6315065c8 P ece7026890434da59563ef0fd2e0d88a: Deleting consensus metadata
I20260809 05:23:25.916828 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 7435
W20260809 05:23:25.938062 7292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:40573 (error 108)
I20260809 05:23:25.938360 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 7666
I20260809 05:23:25.940027 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.939633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45318)
0809 05:23:25.939773 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.939878 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260809 05:23:25.940291 7292 catalog_manager.cc:4584] TS ece7026890434da59563ef0fd2e0d88a (127.22.1.65:40573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e7f7d20198944bc9b8161f6315065c8: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Aug 09 05:23:25 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:23:25.954559 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.939325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.22.1.66:32885)
0809 05:23:25.939503 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:25.939510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:25.939531 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:23:25.940248 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:25.940255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:25.940328 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:25.940419 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:25.941933 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.942729 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.944029 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:25.944415 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:25.944540 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:25.948739 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:25.948755 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:25.948771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:25.948810 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:25.951715 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.952389 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.952397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.952399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.952528 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:25.952887 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:25.952891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:25.952893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:25.953419 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:25.953710 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:25.954015 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 05:23:25.954223 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:23:25.955209 7877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:23:25.994268 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.993687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45320)
0809 05:23:25.993931 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.994065 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260809 05:23:25.995462 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:25.995053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45334)
0809 05:23:25.995205 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:25.995295 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 05:23:26.046737 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 7666
I20260809 05:23:26.056509 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.056002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45336)
0809 05:23:26.056255 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:26.056337 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260809 05:23:26.057610 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.057273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45338)
0809 05:23:26.057418 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:26.057494 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 05:23:26.079699 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 7896
Aug 09 05:23:26 dist-test-slave-rmhg krb5kdc[6555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:23:26.095118 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.080490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43564 (local address 127.22.1.67:44325)
0809 05:23:26.080699 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 05:23:26.080705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:23:26.080728 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:23:26.080868 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:23:26.080873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:23:26.080937 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:23:26.081039 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:23:26.082337 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:26.083158 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:26.084403 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:23:26.084863 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:23:26.085005 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:23:26.089204 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:23:26.089219 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:23:26.089236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:23:26.089281 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:23:26.092261 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:26.092981 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:26.092988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:26.092990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:26.093144 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:23:26.093543 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:23:26.093548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:23:26.093552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:23:26.093913 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:23:26.094274 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0809 05:23:26.094536 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 05:23:26.094758 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:23:26.095749 8083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:23:26.138689 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.138181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45342)
0809 05:23:26.138401 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:26.138511 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 05:23:26.139726 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.139362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45356)
0809 05:23:26.139541 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:26.139612 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260809 05:23:26.217914 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 7896
I20260809 05:23:26.244398 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 7260
I20260809 05:23:26.245656 7354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:23:26.357630 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:23:26.357112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40573 (local address 127.0.0.1:45366)
0809 05:23:26.357310 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0809 05:23:26.357396 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:40573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 05:23:26.357748 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 7260
I20260809 05:23:26.394626 22533 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 05:23:26.731671 22533 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 05:23:26.731948 22533 mini_ranger.cc:66] Stopping Ranger...
I20260809 05:23:27.077540 22533 mini_ranger.cc:68] Stopped Ranger
2026-08-09 05:23:27.077 UTC [6603] LOG: received smart shutdown request
2026-08-09 05:23:27.081 UTC [6603] LOG: background worker "logical replication launcher" (PID 6625) exited with exit code 1
2026-08-09 05:23:27.082 UTC [6614] LOG: shutting down
2026-08-09 05:23:27.233 UTC [6603] LOG: database system is shut down
2026-08-09T05:23:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49661 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:23:27 dist-test-slave-rmhg krb5kdc[8188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:23:27 dist-test-slave-rmhg krb5kdc[8188](info): set up 2 sockets
Aug 09 05:23:27 dist-test-slave-rmhg krb5kdc[8188](info): commencing operation
krb5kdc: starting...
W20260809 05:23:28.578718 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:23:28 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:23:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangeradmin_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangeradmin_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangerlookup_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangerlookup_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
I20260809 05:23:28.940524 22533 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGKhvF4/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/postgres -l logfile start
2026-08-09 05:23:31.241 UTC [8239] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-09 05:23:31.241 UTC [8239] LOG: listening on IPv4 address "127.22.1.84", port 39269
2026-08-09 05:23:31.244 UTC [8239] LOG: listening on Unix socket "/tmp/.s.PGSQL.39269"
2026-08-09 05:23:31.266 UTC [8244] LOG: database system was shut down at 2026-08-09 05:23:29 UTC
2026-08-09 05:23:31.270 UTC [8239] LOG: database system is ready to accept connections
I20260809 05:23:32.343770 22533 mini_postgres.cc:96] Postgres bound to 39269
2026-08-09 05:23:32.364 UTC [8253] FATAL: database "slave" does not exist
127.22.1.84:39269 - accepting connections
I20260809 05:23:32.364635 22533 mini_ranger.cc:161] Starting Ranger...
I20260809 05:23:32.393082 22533 mini_ranger.cc:84] Created miniranger Postgres user
I20260809 05:23:32.808660 22533 mini_ranger.cc:87] Created ranger Postgres database
I20260809 05:23:32.808779 22533 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin
2026-08-09 05:23:33,128 [I] DB FLAVOR :POSTGRES
2026-08-09 05:23:33,128 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 05:23:33,128 [I] Checking connection..
2026-08-09 05:23:33,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-09 05:23:33,398 [I] Checking connection passed.
2026-08-09 05:23:33,398 [I] --------- Verifying version history table ---------
2026-08-09 05:23:33,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:23:33,646 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 05:23:33,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:23:33,901 [I] Table x_db_version_h does not exist in database ranger
2026-08-09 05:23:33,902 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-09 05:23:33,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-09 05:23:34.161 UTC [8348] WARNING: there is no transaction in progress
2026-08-09 05:23:34,169 [I] create_dbversion_catalog.sql file imported successfully
2026-08-09 05:23:34,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-09 05:23:34,421 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-09 05:23:34,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 05:23:34,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-09 05:23:34,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-09 05:23:34,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-rmhg.c.gcp-upstream.internal','N') ;"
2026-08-09 05:23:35,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 05:23:35,424 [I] Table x_portal_user does not exist in database ranger
2026-08-09 05:23:35,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 05:23:35,686 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-09 05:23:35,687 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 05:23:35,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 05:23:36,182 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-09 05:23:36,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-09 05:23:37.106 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.122 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.138 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.153 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.171 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.191 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.252 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.258 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.272 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.283 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.293 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.307 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.316 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.322 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.329 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37.335 UTC [8516] WARNING: there is no transaction in progress
2026-08-09 05:23:37,938 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-09 05:23:37,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-09 05:23:38,175 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-09 05:23:38,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-09 05:23:38,433 [I] Table x_portal_user already exists in database 'ranger'
2026-08-09 05:23:38,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-09 05:23:38,691 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-09 05:23:38,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-09 05:23:38,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-09 05:23:39,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-rmhg.c.gcp-upstream.internal';"
2026-08-09 05:23:39,427 [I] CORE_DB_SCHEMA import status has been updated
2026-08-09 05:23:39,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-09 05:23:39,693 [I] DB_PATCHES have already been applied
I20260809 05:23:39.699012 22533 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGKhvF4/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260809 05:23:39.699103 22533 mini_ranger.cc:193] Using host: 127.22.1.84
I20260809 05:23:39.717118 22533 mini_ranger.cc:227] Ranger admin URL: http://127.22.1.84:58209
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-admin/ews/webapp].
Aug 09, 2026 5:23:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260809 05:23:40.923679 22533 mini_ranger.cc:160] Time spent starting Ranger: real 8.559s user 0.005s sys 0.048s
Aug 09, 2026 5:24:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 09, 2026 5:24:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 09, 2026 5:24:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 09, 2026 5:24:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 09, 2026 5:24:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260809 05:24:03.078953 22533 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.1.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.1.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangerkms_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.1.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/rangerkms_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.1.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.1.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.1.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/HTTP_127.22.1.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 09 05:24:03 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260809 05:24:03.274694 22533 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260809 05:24:03.298861 22533 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260809 05:24:04.100700 22533 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260809 05:24:04.100800 22533 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms
I20260809 05:24:04.100829 22533 mini_ranger_kms.cc:227] Using postgres at 127.22.1.84:39269
2026-08-09 05:24:04,434 [I] DB FLAVOR :POSTGRES
2026-08-09 05:24:04,435 [I] --------- Verifying Ranger DB connection ---------
2026-08-09 05:24:04,435 [I] Checking connection
2026-08-09 05:24:04,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-09 05:24:04,672 [I] connection success
2026-08-09 05:24:04,672 [I] --------- Verifying Ranger DB tables ---------
2026-08-09 05:24:04,672 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-09 05:24:04,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-09 05:24:04,936 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-09 05:24:04,937 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-09 05:24:04,937 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-09 05:24:04,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.1.84:39269/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-09 05:24:05,201 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260809 05:24:05.682055 22533 mini_ranger_kms.cc:310] Created kms service
I20260809 05:24:05.888433 22533 mini_ranger_kms.cc:326] Created kudu user
I20260809 05:24:05.931416 22533 mini_ranger_kms.cc:343] Created rangerkms user
Aug 09 05:24:05 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253043, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:06.305486 22533 mini_ranger_kms.cc:384] Added ranger policy
I20260809 05:24:06.305831 22533 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGKhvF4/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGKhvF4/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260809 05:24:06.305889 22533 mini_ranger_kms.cc:240] Using host: 127.22.1.84
I20260809 05:24:06.327404 22533 mini_ranger_kms.cc:276] Ranger KMS PID: 8813
I20260809 05:24:06.327560 22533 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.1.84:57247
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-kms/ews/webapp].
Aug 09, 2026 5:24:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 09 05:24:08 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786253048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:08.786509 22533 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.512s user 0.077s sys 0.292s
I20260809 05:24:08.786787 22533 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260809 05:24:08.786964 22533 mini_ranger_kms.cc:401] 127.22.1.84:57247/kms/v1/keys
Aug 09 05:24:09 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
Aug 09 05:24:09 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253048, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.1.84@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
Aug 09 05:24:11 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:24:11.857311 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:40971
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:40971
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:12.209134 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:12.209453 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:12.209548 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:12.216634 8900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:12.216754 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:12.216799 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:12.216856 8900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:12.216902 8900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:12.227291 8900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.1.84:57247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:40971
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:12.228683 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:12.230065 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:12.242149 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:12.242149 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:12.242193 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:12.242782 8900 server_base.cc:1034] running on GCE node
I20260809 05:24:12.243660 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:12.245301 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:12.246532 8900 hybrid_clock.cc:648] HybridClock initialized: now 1786253052246476 us; error 49 us; skew 500 ppm
I20260809 05:24:12.246886 8900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:12 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253052, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:12.250082 8900 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:12.251557 8900 webserver.cc:460] Webserver started at http://127.22.1.126:39385/ using document root <none> and password file <none>
I20260809 05:24:12.252219 8900 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:12.252333 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:12.252638 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:24:12 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253052, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:12.311121 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8"
format_stamp: "Formatted at 2026-08-09 05:24:12 on dist-test-slave-rmhg"
server_key: "2955a78325a05fdf66f7b8bc32a0e79c"
server_key_iv: "8d467cc683518b1c33095f3dafbd9d72"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:12.312258 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8"
format_stamp: "Formatted at 2026-08-09 05:24:12 on dist-test-slave-rmhg"
server_key: "2955a78325a05fdf66f7b8bc32a0e79c"
server_key_iv: "8d467cc683518b1c33095f3dafbd9d72"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:12.319587 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260809 05:24:12.337782 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:12.339382 8900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 05:24:12.339633 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8"
format_stamp: "Formatted at 2026-08-09 05:24:12 on dist-test-slave-rmhg"
server_key: "2955a78325a05fdf66f7b8bc32a0e79c"
server_key_iv: "8d467cc683518b1c33095f3dafbd9d72"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:12.339915 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:12.364676 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:12.368422 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:12.368748 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:12.391312 8900 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:40971
I20260809 05:24:12.391336 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:40971 every 8 connection(s)
I20260809 05:24:12.392760 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:12.396661 9003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:12.401322 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 8900
I20260809 05:24:12.401569 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:24:12.402423 22533 external_mini_cluster.cc:1428] Setting key 037f8da90f8a75f54cdd9296188acdb6
I20260809 05:24:12.409102 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Bootstrap starting.
I20260809 05:24:12.412807 9003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:12.414084 9003 log.cc:824] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:12.417501 9003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: No bootstrap required, opened a new log
Aug 09 05:24:12 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:12.425302 9003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:12.425676 9003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:12.425783 9003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cc8efd5b0c484c85486fc5d6bf8ff8, State: Initialized, Role: FOLLOWER
I20260809 05:24:12.425908 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:12.404797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.22.1.126:40971)
0809 05:24:12.405682 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:12.405695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:12.405749 (+ 54us) server_negotiation.cc:407] Connection header received
0809 05:24:12.407604 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:12.407632 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:12.408076 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:12.408426 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:12.410235 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:12.411584 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:12.413416 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:12.414127 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:12.414300 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:12.418655 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:12.418664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:12.418680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:12.418717 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:12.421758 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:12.422435 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:12.422442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:12.422450 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:12.422611 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:12.422997 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:12.423001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:12.423003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:12.423575 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:12.423875 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:12.424363 (+ 488us) server_negotiation.cc:299] Negotiation successful
0809 05:24:12.424778 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":424,"threads_started":1}
I20260809 05:24:12.426404 9003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:12.426651 9003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:12.426787 9003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:12.426956 9003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:12.429363 9003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:12.429838 9003 leader_election.cc:302] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93cc8efd5b0c484c85486fc5d6bf8ff8; no voters:
I20260809 05:24:12.430469 9003 leader_election.cc:288] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:24:12.430565 9008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:12.431258 9008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 LEADER]: Becoming Leader. State: Replica: 93cc8efd5b0c484c85486fc5d6bf8ff8, State: Running, Role: LEADER
I20260809 05:24:12.431799 9008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:12.432467 9003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:12.434409 9010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cc8efd5b0c484c85486fc5d6bf8ff8. Latest consensus state: current_term: 1 leader_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } } }
I20260809 05:24:12.434791 9010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:12.435320 9009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } } }
I20260809 05:24:12.435676 9009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:12.437952 9017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:12.443440 9017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:12.444173 9003 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 05:24:12.463099 9017 catalog_manager.cc:1347] Generated new cluster ID: a1f61a27e85a4ddbaba5a3846ead18ff
I20260809 05:24:12.463263 9017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:12.500890 9017 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:24:12.503003 9017 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:12.520102 9017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Generated new TSK 0
I20260809 05:24:12.521466 9017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:24:13.232110 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:13 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260809 05:24:13.684568 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:13.685133 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:13.685380 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:13.696949 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:13.697110 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:13.697316 9072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:13.713089 9072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:13.715214 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:13.717115 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:13.734210 9086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:13.735677 9087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:13.741201 9091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:13.741559 9072 server_base.cc:1034] running on GCE node
I20260809 05:24:13.742599 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:13.744052 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:13.745378 9072 hybrid_clock.cc:648] HybridClock initialized: now 1786253053745273 us; error 91 us; skew 500 ppm
I20260809 05:24:13.745795 9072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:13 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:13.750283 9072 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:13.769418 9072 webserver.cc:460] Webserver started at http://127.22.1.65:44033/ using document root <none> and password file <none>
I20260809 05:24:13.770460 9072 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:13.770654 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:13.771044 9072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:24:13 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:13.791256 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "222ab1309f2549d0ab18039d4fa95e15"
format_stamp: "Formatted at 2026-08-09 05:24:13 on dist-test-slave-rmhg"
server_key: "4072ddddc98301bfdd977bef3e1732c1"
server_key_iv: "3333ecc4262a3ee6c220d361da0f7dc5"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:13.792407 9072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "222ab1309f2549d0ab18039d4fa95e15"
format_stamp: "Formatted at 2026-08-09 05:24:13 on dist-test-slave-rmhg"
server_key: "4072ddddc98301bfdd977bef3e1732c1"
server_key_iv: "3333ecc4262a3ee6c220d361da0f7dc5"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:13.798746 9072 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260809 05:24:13.815673 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:13.818344 9072 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 05:24:13.818542 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "222ab1309f2549d0ab18039d4fa95e15"
format_stamp: "Formatted at 2026-08-09 05:24:13 on dist-test-slave-rmhg"
server_key: "4072ddddc98301bfdd977bef3e1732c1"
server_key_iv: "3333ecc4262a3ee6c220d361da0f7dc5"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:13.818792 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:13.903890 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:13.909722 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:13.910192 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:13.917809 9072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:13.926455 9072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:13.926755 9072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260809 05:24:13.927534 9072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:13.927696 9072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
Aug 09 05:24:14 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:14.408198 9072 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:43265
I20260809 05:24:14.408288 9269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:43265 every 8 connection(s)
I20260809 05:24:14.409940 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:14.414673 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 9072
I20260809 05:24:14.414893 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:24:14.415743 22533 external_mini_cluster.cc:1428] Setting key 6a58f7f7e3a92b95f7bd51c5143d18eb
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Aug 09 05:24:14 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:14.473707 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:14.425135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:53467 (local address 127.22.1.126:40971)
0809 05:24:14.450774 (+ 25639us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:14.450781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:14.453262 (+ 2481us) server_negotiation.cc:407] Connection header received
0809 05:24:14.454922 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:14.454928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:14.455033 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:14.455118 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:14.457641 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:14.458696 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:14.461016 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:14.461638 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:14.461732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:14.466205 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:14.466212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:14.466217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:14.466271 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:14.469630 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:14.470657 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:14.470663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:14.470666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:14.470833 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:14.471443 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:14.471448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:14.471451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:14.471833 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:14.471977 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:14.473054 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0809 05:24:14.473331 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25474,"thread_start_us":25640,"threads_started":1}
I20260809 05:24:14.476656 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:14.424486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.65:53467)
0809 05:24:14.451636 (+ 27150us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:14.451713 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:14.454565 (+ 2852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:14.455712 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:14.455724 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:14.456403 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:14.457408 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:14.457438 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:14.458913 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:14.458919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:14.460836 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:14.460851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:14.462155 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:14.462164 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:14.462442 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:14.463239 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:14.463268 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:14.465938 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:14.469828 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:14.469838 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:14.469858 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:14.470501 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:14.471008 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:14.471013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:14.471017 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:14.471297 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:14.472089 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:14.472097 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:14.472561 (+ 464us) client_negotiation.cc:769] Sending connection context
0809 05:24:14.474542 (+ 1981us) client_negotiation.cc:240] Negotiation successful
0809 05:24:14.475178 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26730,"thread_start_us":29187,"threads_started":1}
I20260809 05:24:14.478869 9270 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:14.479147 9270 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:14.479993 9270 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:14.482744 8949 ts_manager.cc:194] Registered new tserver with Master: 222ab1309f2549d0ab18039d4fa95e15 (127.22.1.65:43265)
I20260809 05:24:14.484766 8949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:53467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:24:14.502751 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:14 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253053, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
W20260809 05:24:14.936044 9278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:14.936352 9278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:14.936482 9278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:14.944056 9278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:14.944192 9278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:14.944370 9278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:14.954598 9278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:14.955981 9278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:14.957403 9278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:14.972003 9297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:14.972078 9298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:14.972638 9301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:14.973744 9278 server_base.cc:1034] running on GCE node
I20260809 05:24:14.974489 9278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:14.975704 9278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:14.976948 9278 hybrid_clock.cc:648] HybridClock initialized: now 1786253054976881 us; error 53 us; skew 500 ppm
I20260809 05:24:14.977355 9278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:14 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253054, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:14.981387 9278 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:14.983162 9278 webserver.cc:460] Webserver started at http://127.22.1.66:42555/ using document root <none> and password file <none>
I20260809 05:24:14.984082 9278 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:14.984233 9278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:14.984627 9278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:24:14 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253054, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:15.005061 9278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "50dcbcacb85741a0966aac7200c358bc"
format_stamp: "Formatted at 2026-08-09 05:24:14 on dist-test-slave-rmhg"
server_key: "1c777d808a93b7eb2235f6f656e0a19c"
server_key_iv: "82c1b0edc24a346a75054ead0b693baf"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.006922 9278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "50dcbcacb85741a0966aac7200c358bc"
format_stamp: "Formatted at 2026-08-09 05:24:14 on dist-test-slave-rmhg"
server_key: "1c777d808a93b7eb2235f6f656e0a19c"
server_key_iv: "82c1b0edc24a346a75054ead0b693baf"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.014484 9278 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260809 05:24:15.035476 9311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.037452 9278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:24:15.037734 9278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "50dcbcacb85741a0966aac7200c358bc"
format_stamp: "Formatted at 2026-08-09 05:24:14 on dist-test-slave-rmhg"
server_key: "1c777d808a93b7eb2235f6f656e0a19c"
server_key_iv: "82c1b0edc24a346a75054ead0b693baf"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.038004 9278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:15.077116 9278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:15.082691 9278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:15.083045 9278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:15.084290 9278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:15.086170 9278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:15.086263 9278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.086392 9278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:15.086452 9278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.118724 9278 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:46129
I20260809 05:24:15.118763 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:46129 every 8 connection(s)
I20260809 05:24:15.119992 9278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:15.129585 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 9278
I20260809 05:24:15.129729 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:24:15.130295 22533 external_mini_cluster.cc:1428] Setting key 365d57aaa0b99dc1081fdcdc7cca8bb6
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253054, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:15.140130 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.122547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:38967 (local address 127.22.1.126:40971)
0809 05:24:15.122970 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:15.122974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:15.123948 (+ 974us) server_negotiation.cc:407] Connection header received
0809 05:24:15.125858 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:15.125864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:15.125941 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:15.126041 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:15.127524 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.128278 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.130068 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.130512 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.130630 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.134329 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:15.134335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:15.134338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:15.134371 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:15.136944 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.137778 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.137783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.137787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.137908 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.138332 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.138336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.138338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.138634 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:15.138804 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:15.139547 (+ 743us) server_negotiation.cc:299] Negotiation successful
0809 05:24:15.139774 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":223,"threads_started":1}
I20260809 05:24:15.141146 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.123009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.66:38967)
0809 05:24:15.123777 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:15.123817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:15.125584 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:15.126295 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:15.126303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:15.126728 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:15.127338 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:15.127357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.128435 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.128439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:15.129905 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:15.129930 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.130775 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.130785 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:15.131114 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.131781 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:15.131804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:15.134112 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:15.137083 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:15.137092 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:15.137133 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:15.137615 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:15.138017 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:15.138022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:15.138026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:15.138210 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:15.138800 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:15.138810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:15.139150 (+ 340us) client_negotiation.cc:769] Sending connection context
0809 05:24:15.139396 (+ 246us) client_negotiation.cc:240] Negotiation successful
0809 05:24:15.139929 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":323,"threads_started":1}
I20260809 05:24:15.142778 9492 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:15.143101 9492 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:15.144011 9492 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:15.145645 8949 ts_manager.cc:194] Registered new tserver with Master: 50dcbcacb85741a0966aac7200c358bc (127.22.1.66:46129)
I20260809 05:24:15.146703 8949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:38967
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:24:15.188372 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:15.488893 9270 heartbeater.cc:503] Master 127.22.1.126:40971 was elected leader, sending a full tablet report...
W20260809 05:24:15.526994 9502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:15.527287 9502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:15.527383 9502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:15.534768 9502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:15.534884 9502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:15.535008 9502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:15.545063 9502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:15.546396 9502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:15.547694 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:15.560637 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:15.560674 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:15.560714 9510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:15.562067 9502 server_base.cc:1034] running on GCE node
I20260809 05:24:15.562716 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:15.563732 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:15.564944 9502 hybrid_clock.cc:648] HybridClock initialized: now 1786253055564877 us; error 56 us; skew 500 ppm
I20260809 05:24:15.565346 9502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:15.568567 9502 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:15.570087 9502 webserver.cc:460] Webserver started at http://127.22.1.67:39185/ using document root <none> and password file <none>
I20260809 05:24:15.570783 9502 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:15.570888 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:15.571166 9502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:15.586455 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "52327e14a8364090a96b97b8d07f7bd5"
format_stamp: "Formatted at 2026-08-09 05:24:15 on dist-test-slave-rmhg"
server_key: "e54d98edc314acf5f13fb42f6b552a4d"
server_key_iv: "98151ce0b69763d478fb22f2d7865d56"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.587299 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "52327e14a8364090a96b97b8d07f7bd5"
format_stamp: "Formatted at 2026-08-09 05:24:15 on dist-test-slave-rmhg"
server_key: "e54d98edc314acf5f13fb42f6b552a4d"
server_key_iv: "98151ce0b69763d478fb22f2d7865d56"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.592808 9502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260809 05:24:15.609927 9523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.611471 9502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:15.611639 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "52327e14a8364090a96b97b8d07f7bd5"
format_stamp: "Formatted at 2026-08-09 05:24:15 on dist-test-slave-rmhg"
server_key: "e54d98edc314acf5f13fb42f6b552a4d"
server_key_iv: "98151ce0b69763d478fb22f2d7865d56"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:15.611864 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:15.634500 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:15.638319 9502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:15.638608 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:15.639593 9502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:15.641027 9502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:15.641104 9502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.641245 9502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:15.641296 9502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:15.671633 9502 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40815
I20260809 05:24:15.671676 9704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40815 every 8 connection(s)
I20260809 05:24:15.672933 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:15.675083 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 9502
I20260809 05:24:15.675246 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:24:15.675781 22533 external_mini_cluster.cc:1428] Setting key cf67b2c7e93e86dfdb159e05417f0067
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253055, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:15.692617 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.675744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:34007 (local address 127.22.1.126:40971)
0809 05:24:15.676621 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:15.676627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:15.677409 (+ 782us) server_negotiation.cc:407] Connection header received
0809 05:24:15.679156 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:15.679159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:15.679223 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:15.679309 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:15.680619 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.681396 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.682965 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.683385 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.683476 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.686666 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:15.686672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:15.686675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:15.686708 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:15.689352 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.690298 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.690304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.690307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.690450 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.690928 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.690932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.690934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.691303 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:15.691421 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:15.692159 (+ 738us) server_negotiation.cc:299] Negotiation successful
0809 05:24:15.692344 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":579,"threads_started":1}
I20260809 05:24:15.693658 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.676321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.67:34007)
0809 05:24:15.677232 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:15.677289 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:15.678918 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:15.679485 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:15.679493 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:15.679907 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:15.680484 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:15.680502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.681565 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.681569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:15.682849 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:15.682868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.683600 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.683605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:15.683810 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.684394 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:15.684416 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:15.686420 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:15.689503 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:15.689513 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:15.689530 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:15.690180 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:15.690566 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:15.690571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:15.690574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:15.690812 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:15.691429 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:15.691436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:15.691771 (+ 335us) client_negotiation.cc:769] Sending connection context
0809 05:24:15.692033 (+ 262us) client_negotiation.cc:240] Negotiation successful
0809 05:24:15.692486 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":350,"threads_started":1}
I20260809 05:24:15.695027 9709 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:15.695318 9709 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:15.695997 9709 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:15.697407 8949 ts_manager.cc:194] Registered new tserver with Master: 52327e14a8364090a96b97b8d07f7bd5 (127.22.1.67:40815)
I20260809 05:24:15.698194 8949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:34007
I20260809 05:24:15.703590 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:24:15.705264 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 9072
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:24:15.721926 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.706239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.22.1.65:43265)
0809 05:24:15.706876 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:15.706883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:15.706908 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:24:15.706970 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:15.706977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:15.707172 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:15.707299 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:15.708744 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.709883 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.711414 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.711992 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.712095 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.716342 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:15.716354 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:15.716369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:15.716404 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:15.719224 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.719862 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.719870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.719872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.720018 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.720334 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.720336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.720338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.720683 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:15.720928 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:15.721325 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 05:24:15.721588 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":349,"threads_started":1}
I20260809 05:24:15.723181 9263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:15.807096 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 9072
I20260809 05:24:15.827833 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 9278
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:24:15.844513 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.828838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.22.1.66:46129)
0809 05:24:15.829624 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:15.829634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:15.829665 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:24:15.829728 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:15.829734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:15.829967 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:15.830103 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:15.831361 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.832416 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.834008 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.834609 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.834716 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.838772 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:15.838785 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:15.838801 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:15.838838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:15.841521 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.842211 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.842217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.842219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.842387 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.842699 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.842703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.842705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.843070 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:15.843313 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:15.843610 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 05:24:15.843865 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":411,"threads_started":1}
I20260809 05:24:15.846052 9481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:15.927641 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 9278
I20260809 05:24:15.949736 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 9502
Aug 09 05:24:15 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:24:15.966069 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:15.950732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.22.1.67:40815)
0809 05:24:15.951407 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:15.951415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:15.951446 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:24:15.951508 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:15.951516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:15.951735 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:15.951870 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:15.953229 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.954307 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.955698 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:15.956287 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:15.956389 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:15.960601 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:15.960618 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:15.960635 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:15.960673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:15.963349 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.964063 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.964070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.964072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.964203 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:15.964553 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:15.964556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:15.964558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:15.964898 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:15.965145 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:15.965556 (+ 411us) server_negotiation.cc:299] Negotiation successful
0809 05:24:15.965765 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":352,"threads_started":1}
I20260809 05:24:15.967391 9697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:16.049002 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 9502
I20260809 05:24:16.067531 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 8900
I20260809 05:24:16.068874 8996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:16.174535 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 8900
I20260809 05:24:16.200788 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:40971
--webserver_interface=127.22.1.126
--webserver_port=39385
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:40971
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:16.537372 9726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:16.537690 9726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:16.537806 9726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:16.545547 9726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:16.545671 9726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:16.545742 9726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:16.545820 9726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:16.545882 9726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:16.556267 9726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:40971
--ranger_config_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=39385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:16.557682 9726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:16.559078 9726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:16.571300 9736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:16.571244 9738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:16.571278 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:16.571833 9726 server_base.cc:1034] running on GCE node
I20260809 05:24:16.572911 9726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:16.574486 9726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:16.575690 9726 hybrid_clock.cc:648] HybridClock initialized: now 1786253056575643 us; error 41 us; skew 500 ppm
I20260809 05:24:16.576050 9726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:16 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:16.579291 9726 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:16.580945 9726 webserver.cc:460] Webserver started at http://127.22.1.126:39385/ using document root <none> and password file <none>
I20260809 05:24:16.581728 9726 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:16.581866 9726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 05:24:16 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253056, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:16.622439 9726 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:24:16.625912 9752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:16.627663 9726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 05:24:16.627874 9726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8"
format_stamp: "Formatted at 2026-08-09 05:24:12 on dist-test-slave-rmhg"
server_key: "2955a78325a05fdf66f7b8bc32a0e79c"
server_key_iv: "8d467cc683518b1c33095f3dafbd9d72"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:16.628903 9726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:16.647398 9726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:16.651446 9726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:16.651757 9726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:16.669901 9726 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:40971
I20260809 05:24:16.669924 9823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:40971 every 8 connection(s)
I20260809 05:24:16.671514 9726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:16.678063 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 9726
I20260809 05:24:16.678805 9826 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:24:16.679165 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:43265
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:40971
--webserver_port=44033
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:16.683389 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Bootstrap starting.
I20260809 05:24:16.703862 9826 log.cc:824] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:16.713650 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:24:16.714201 9826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Bootstrap complete.
I20260809 05:24:16.723328 9826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:16.724213 9826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cc8efd5b0c484c85486fc5d6bf8ff8, State: Initialized, Role: FOLLOWER
I20260809 05:24:16.724931 9826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:16.725203 9826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:16.725328 9826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:16.725502 9826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:24:16.729585 9826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:16.730149 9826 leader_election.cc:302] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93cc8efd5b0c484c85486fc5d6bf8ff8; no voters:
I20260809 05:24:16.730789 9826 leader_election.cc:288] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 05:24:16.730933 9832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 05:24:16.731990 9832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [term 2 LEADER]: Becoming Leader. State: Replica: 93cc8efd5b0c484c85486fc5d6bf8ff8, State: Running, Role: LEADER
I20260809 05:24:16.732666 9832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } }
I20260809 05:24:16.733079 9826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:16.735241 9834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cc8efd5b0c484c85486fc5d6bf8ff8. Latest consensus state: current_term: 2 leader_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } } }
I20260809 05:24:16.736140 9834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:16.735807 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc8efd5b0c484c85486fc5d6bf8ff8" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 40971 } } }
I20260809 05:24:16.736545 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:16.746613 9841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:16.753058 9841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:16.753996 9826 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260809 05:24:16.754791 9841 catalog_manager.cc:1259] Loaded cluster ID: a1f61a27e85a4ddbaba5a3846ead18ff
I20260809 05:24:16.754969 9841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:16.758986 9841 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:16.760546 9841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Loaded TSK: 0
I20260809 05:24:16.762332 9841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 05:24:17.141364 9829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:17.141945 9829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:17.142195 9829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:17.153923 9829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:17.154148 9829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:17.154354 9829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:17.170321 9829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:43265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=44033
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:17.172430 9829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:17.174415 9829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:17.193858 9881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:17.194941 9882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:17.214986 9885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:17.215535 9829 server_base.cc:1034] running on GCE node
I20260809 05:24:17.216473 9829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:17.218130 9829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:17.219482 9829 hybrid_clock.cc:648] HybridClock initialized: now 1786253057219406 us; error 66 us; skew 500 ppm
I20260809 05:24:17.219920 9829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:17 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:17.228276 9829 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:17.241392 9829 webserver.cc:460] Webserver started at http://127.22.1.65:44033/ using document root <none> and password file <none>
I20260809 05:24:17.242408 9829 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:17.242586 9829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 05:24:17 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:17.273561 9829 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:24:17.277335 9899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:17.279019 9829 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 05:24:17.279207 9829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "222ab1309f2549d0ab18039d4fa95e15"
format_stamp: "Formatted at 2026-08-09 05:24:13 on dist-test-slave-rmhg"
server_key: "4072ddddc98301bfdd977bef3e1732c1"
server_key_iv: "3333ecc4262a3ee6c220d361da0f7dc5"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:17.280651 9829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:17.309744 9829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:17.314085 9829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:17.314376 9829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:17.315503 9829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:17.317029 9829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:17.317132 9829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:17.317266 9829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:17.317358 9829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:17.402276 9829 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:43265
I20260809 05:24:17.402329 10078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:43265 every 8 connection(s)
I20260809 05:24:17.403784 9829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:17.407713 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 9829
I20260809 05:24:17.408525 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:46129
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:40971
--webserver_port=42555
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:17 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:17.460932 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:17.419859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:53633 (local address 127.22.1.126:40971)
0809 05:24:17.432735 (+ 12876us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:17.432760 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:17.433960 (+ 1200us) server_negotiation.cc:407] Connection header received
0809 05:24:17.437176 (+ 3216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:17.437203 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:17.437826 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:17.438327 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:17.440469 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:17.442533 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:17.445136 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:17.446254 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:17.446435 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:17.451581 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:17.451592 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:17.451615 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:17.451669 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:17.455559 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:17.456557 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:17.456566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:17.456575 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:17.456800 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:17.457415 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:17.457419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:17.457422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:17.458061 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:17.458248 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:17.459306 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0809 05:24:17.459736 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12519,"thread_start_us":12267,"threads_started":1}
I20260809 05:24:17.460942 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:17.419986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.65:53633)
0809 05:24:17.433529 (+ 13543us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:17.433599 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:17.436289 (+ 2690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:17.438571 (+ 2282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:17.438584 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:17.439291 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:17.440186 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:17.440219 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:17.442762 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:17.442769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:17.444949 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:17.444967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:17.446840 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:17.446849 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:17.447120 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:17.448322 (+ 1202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:17.448355 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:17.451231 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:17.455757 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:17.455767 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:17.455784 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:17.456363 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:17.456969 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:17.456975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:17.456979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:17.457285 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:17.458243 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:17.458252 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:17.458690 (+ 438us) client_negotiation.cc:769] Sending connection context
0809 05:24:17.459041 (+ 351us) client_negotiation.cc:240] Negotiation successful
0809 05:24:17.459620 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13101,"thread_start_us":12873,"threads_started":1}
I20260809 05:24:17.465298 10080 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:17.465662 10080 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:17.466557 10080 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
W20260809 05:24:17.470130 10080 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:40971 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 05:24:17.470378 10080 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:17.470685 10080 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:17.472203 10080 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:17.472607 10080 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
W20260809 05:24:17.473873 10080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 05:24:17.865917 10082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:17.866233 10082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:17.866376 10082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:17.877547 10082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:17.877770 10082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:17.877948 10082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:17.892202 10082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:46129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=42555
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:17.893939 10082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:17.895633 10082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:17.913247 10112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:17.917779 10111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:17.920863 10115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:17.921739 10082 server_base.cc:1034] running on GCE node
I20260809 05:24:17.922787 10082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:17.933472 10082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:17.936992 10082 hybrid_clock.cc:648] HybridClock initialized: now 1786253057936887 us; error 94 us; skew 500 ppm
I20260809 05:24:17.937543 10082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:17 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:17.942293 10082 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:17.949740 10082 webserver.cc:460] Webserver started at http://127.22.1.66:42555/ using document root <none> and password file <none>
I20260809 05:24:17.950778 10082 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:17.950955 10082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 05:24:17 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:18.017050 10082 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.008s sys 0.012s
I20260809 05:24:18.021422 10132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:18.023492 10082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260809 05:24:18.023778 10082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "50dcbcacb85741a0966aac7200c358bc"
format_stamp: "Formatted at 2026-08-09 05:24:14 on dist-test-slave-rmhg"
server_key: "1c777d808a93b7eb2235f6f656e0a19c"
server_key_iv: "82c1b0edc24a346a75054ead0b693baf"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:18.025413 10082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:18.069212 10082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:18.075786 10082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:18.076191 10082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:18.077418 10082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:18.079114 10082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:18.079222 10082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:18.079329 10082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:18.079397 10082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:18.126248 10082 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:46129
I20260809 05:24:18.126325 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:46129 every 8 connection(s)
I20260809 05:24:18.128374 10082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:18.133062 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 10082
I20260809 05:24:18.133860 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.1.84:57247/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:40815
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:40971
--webserver_port=39185
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253057, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:18.161996 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:18.135840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49243 (local address 127.22.1.126:40971)
0809 05:24:18.144315 (+ 8475us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:18.144321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:18.144347 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:24:18.144418 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:18.144424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:18.144532 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:18.144662 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:18.146881 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.148116 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.150393 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.150801 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.150920 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:18.154415 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:18.154419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:18.154423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:18.154468 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:18.157225 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.158096 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.158102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.158105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.158265 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.158815 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.158820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.158824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.159215 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:18.159371 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:18.161354 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0809 05:24:18.161590 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8324,"thread_start_us":8206,"threads_started":1}
I20260809 05:24:18.163476 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:18.132174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.66:49243)
0809 05:24:18.137843 (+ 5669us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:18.137915 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:18.140375 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:18.145041 (+ 4666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:18.145055 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:18.145733 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:18.146625 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:18.146675 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.148335 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.148341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:18.150225 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:18.150248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.151059 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.151067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:18.151296 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:18.151980 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:18.152004 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:18.154170 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:18.157357 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:18.157368 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:18.157388 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:18.157926 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:18.158404 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:18.158409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:18.158413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:18.158670 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:18.159447 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:18.159455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:18.159903 (+ 448us) client_negotiation.cc:769] Sending connection context
0809 05:24:18.160236 (+ 333us) client_negotiation.cc:240] Negotiation successful
0809 05:24:18.160826 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5321,"thread_start_us":5051,"threads_started":1}
I20260809 05:24:18.165498 10291 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:18.165844 10291 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:18.166775 10291 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:18.168866 9773 ts_manager.cc:194] Registered new tserver with Master: 50dcbcacb85741a0966aac7200c358bc (127.22.1.66:46129)
I20260809 05:24:18.170498 9773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49243
I20260809 05:24:18.474061 10080 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:18.474525 10080 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:18.476039 9773 ts_manager.cc:194] Registered new tserver with Master: 222ab1309f2549d0ab18039d4fa95e15 (127.22.1.65:43265)
I20260809 05:24:18.477399 9773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:53633
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
W20260809 05:24:18.631023 10295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:18.631484 10295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:18.631659 10295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:18.643536 10295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:18.643697 10295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:18.643949 10295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:18.660128 10295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.22.1.84:57247/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:40815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=39185
--tserver_master_addrs=127.22.1.126:40971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:18.662284 10295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:18.663652 10295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:18.696264 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:18.697757 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:18.698429 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:18.698923 10295 server_base.cc:1034] running on GCE node
I20260809 05:24:18.699577 10295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:18.700624 10295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:18.701949 10295 hybrid_clock.cc:648] HybridClock initialized: now 1786253058701859 us; error 57 us; skew 500 ppm
I20260809 05:24:18.702291 10295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:18.705791 10295 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:18.710439 10295 webserver.cc:460] Webserver started at http://127.22.1.67:39185/ using document root <none> and password file <none>
I20260809 05:24:18.711182 10295 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:18.711321 10295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:18.733449 10295 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260809 05:24:18.738840 10318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:18.740299 10295 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:24:18.740480 10295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "52327e14a8364090a96b97b8d07f7bd5"
format_stamp: "Formatted at 2026-08-09 05:24:15 on dist-test-slave-rmhg"
server_key: "e54d98edc314acf5f13fb42f6b552a4d"
server_key_iv: "98151ce0b69763d478fb22f2d7865d56"
server_key_version: "kuduclusterkey@0"
I20260809 05:24:18.741566 10295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:18.764997 10295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:18.769294 10295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:18.769582 10295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:18.770700 10295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:18.772182 10295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:18.772282 10295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:18.772394 10295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:18.772466 10295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for HTTP/127.22.1.84@KRBTEST.COM
I20260809 05:24:18.888959 10295 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:40815
I20260809 05:24:18.889001 10492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:40815 every 8 connection(s)
I20260809 05:24:18.891085 10295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:18.891599 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 10295
I20260809 05:24:18.957137 10496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:18.892681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.22.1.126:40971)
0809 05:24:18.927607 (+ 34926us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:18.927614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:18.927642 (+ 28us) server_negotiation.cc:407] Connection header received
0809 05:24:18.927738 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:18.927744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:18.927840 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:18.930438 (+ 2598us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:18.930779 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.933660 (+ 2881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.935855 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.936553 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.940530 (+ 3977us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:18.940763 (+ 233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:18.940767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:18.940770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:18.940797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:18.951694 (+ 10897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.953846 (+ 2152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.953852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.953856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.954028 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.954533 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.954537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.954540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.954901 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:18.956372 (+ 1471us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:18.956521 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 05:24:18.956756 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34779,"thread_start_us":34651,"threads_started":1}
Aug 09 05:24:18 dist-test-slave-rmhg krb5kdc[8188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253058, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:18.973805 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:18.912135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:40971 (local address 127.22.1.67:37403)
0809 05:24:18.923425 (+ 11290us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:18.923508 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:18.932866 (+ 9358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:18.943742 (+ 10876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:18.943757 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:18.944459 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:18.945500 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:18.945532 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.948551 (+ 3019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.948557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:18.950667 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:18.950685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.957802 (+ 7117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.957812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:18.958174 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:18.959017 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:18.959051 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:18.963746 (+ 4695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:18.968114 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:18.968124 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:18.968147 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:18.968848 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:18.969677 (+ 829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:18.969683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:18.969687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:18.969978 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:18.970772 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:18.970781 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:18.971243 (+ 462us) client_negotiation.cc:769] Sending connection context
0809 05:24:18.971619 (+ 376us) client_negotiation.cc:240] Negotiation successful
0809 05:24:18.972242 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10894,"thread_start_us":13567,"threads_started":1}
I20260809 05:24:18.976516 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:18.912917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:37403 (local address 127.22.1.126:40971)
0809 05:24:18.931548 (+ 18631us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:18.931553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:18.931578 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:24:18.942837 (+ 11259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:18.942844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:18.942950 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:18.943071 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:18.945760 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.948327 (+ 2567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.951864 (+ 3537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:18.952528 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:18.952657 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:18.964062 (+ 11405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:18.964069 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:18.964074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:18.964125 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:18.967878 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.969025 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.969031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.969035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.969529 (+ 494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:18.970134 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:18.970140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:18.970143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:18.970545 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:18.970682 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:18.974944 (+ 4262us) server_negotiation.cc:299] Negotiation successful
0809 05:24:18.976136 (+ 1192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18479,"thread_start_us":18654,"threads_started":1}
I20260809 05:24:18.978549 10494 heartbeater.cc:347] Connected to a master server at 127.22.1.126:40971
I20260809 05:24:18.978896 10494 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:18.979867 10494 heartbeater.cc:511] Master 127.22.1.126:40971 requested a full tablet report, sending...
I20260809 05:24:18.981659 9773 ts_manager.cc:194] Registered new tserver with Master: 52327e14a8364090a96b97b8d07f7bd5 (127.22.1.67:40815)
I20260809 05:24:18.982875 9773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:37403
I20260809 05:24:18.988978 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:24:19.045688 10501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.025656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.22.1.126:40971)
0809 05:24:19.025849 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.025856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.027897 (+ 2041us) server_negotiation.cc:407] Connection header received
0809 05:24:19.028272 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.028279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.028381 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.028466 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:19.030914 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.032051 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.034342 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.035015 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.035138 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.038841 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:19.038846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:19.038849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:19.038891 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:19.042073 (+ 3182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.043075 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.043081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.043085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.043243 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.043725 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.043731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.043734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.044130 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:19.044357 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.045041 (+ 684us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.045327 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:24:19.051594 9773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:24:19.059484 9773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:24:19.144749 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.114394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:46129 (local address 127.0.0.1:55220)
0809 05:24:19.129462 (+ 15068us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:19.129530 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:19.129766 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:19.133315 (+ 3549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:19.133326 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:19.133370 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:19.136830 (+ 3460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.136855 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.139048 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.139054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.141461 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.141482 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.143723 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.143730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.143947 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.144016 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 05:24:19.144150 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 05:24:19.144354 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14708,"threads_started":1}
I20260809 05:24:19.145973 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.114957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.22.1.66:46129)
0809 05:24:19.128732 (+ 13775us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.128744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.129652 (+ 908us) server_negotiation.cc:407] Connection header received
0809 05:24:19.130054 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.130062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.130334 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.130537 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:19.137136 (+ 6599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.138850 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.141683 (+ 2833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.143181 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.143322 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.143567 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.145305 (+ 1738us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.145503 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13543,"thread_start_us":17131,"threads_started":1}
I20260809 05:24:19.156323 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.115256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.22.1.67:40815)
0809 05:24:19.132243 (+ 16987us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.132254 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.145146 (+ 12892us) server_negotiation.cc:407] Connection header received
0809 05:24:19.146512 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.146522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.146817 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.147017 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:19.147750 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.149317 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.152168 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.154877 (+ 2709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.154979 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.155218 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.155815 (+ 597us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.155992 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16784,"thread_start_us":17012,"threads_started":1}
I20260809 05:24:19.155933 10200 tablet_service.cc:1467] Processing CreateTablet for tablet 6820372e358f43ad9e738364f9100682 (DEFAULT_TABLE table=test-table [id=3dcdd10b44b747b294e261c1b87562ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:19.157060 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.115119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40815 (local address 127.0.0.1:37046)
0809 05:24:19.145015 (+ 29896us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:19.145039 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:19.146293 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:19.147364 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:19.147368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:19.147396 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:19.147560 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.147571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.149492 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.149496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.151993 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.152006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.155380 (+ 3374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.155386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.155572 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.155627 (+ 55us) client_negotiation.cc:769] Sending connection context
0809 05:24:19.156547 (+ 920us) client_negotiation.cc:240] Negotiation successful
0809 05:24:19.156725 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29767,"mutex_wait_us":1,"thread_start_us":14872,"threads_started":1}
I20260809 05:24:19.161096 10397 tablet_service.cc:1467] Processing CreateTablet for tablet 6820372e358f43ad9e738364f9100682 (DEFAULT_TABLE table=test-table [id=3dcdd10b44b747b294e261c1b87562ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:19.162793 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820372e358f43ad9e738364f9100682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:19.166108 10200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820372e358f43ad9e738364f9100682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:19.168989 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.151005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43265 (local address 127.0.0.1:43334)
0809 05:24:19.151519 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:19.151537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:19.151653 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:19.153190 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:19.153195 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:19.153222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:19.153573 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.153584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.155577 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.155582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.165937 (+ 10355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.165960 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.168201 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.168207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.168388 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.168423 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 05:24:19.168523 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 05:24:19.168671 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":39}
I20260809 05:24:19.171037 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.151358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.22.1.65:43265)
0809 05:24:19.152320 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.152329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.152362 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:24:19.152465 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.152473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.152741 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.152934 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:19.153790 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.155397 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.166191 (+ 10794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.167678 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.167807 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.168032 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.170495 (+ 2463us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.170698 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":5142,"threads_started":1}
I20260809 05:24:19.175190 9971 tablet_service.cc:1467] Processing CreateTablet for tablet 6820372e358f43ad9e738364f9100682 (DEFAULT_TABLE table=test-table [id=3dcdd10b44b747b294e261c1b87562ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:19.176571 10291 heartbeater.cc:503] Master 127.22.1.126:40971 was elected leader, sending a full tablet report...
I20260809 05:24:19.176759 9971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6820372e358f43ad9e738364f9100682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:19.203370 10539 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Bootstrap starting.
I20260809 05:24:19.206611 10540 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Bootstrap starting.
I20260809 05:24:19.208065 10541 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Bootstrap starting.
I20260809 05:24:19.210529 10540 tablet_bootstrap.cc:654] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:19.211114 10539 tablet_bootstrap.cc:654] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:19.211876 10540 log.cc:824] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:19.212001 10541 tablet_bootstrap.cc:654] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:19.212332 10539 log.cc:824] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:19.213275 10541 log.cc:824] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:19.221724 10540 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: No bootstrap required, opened a new log
I20260809 05:24:19.222046 10540 ts_tablet_manager.cc:1397] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20260809 05:24:19.222435 10541 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: No bootstrap required, opened a new log
I20260809 05:24:19.222741 10541 ts_tablet_manager.cc:1397] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260809 05:24:19.223515 10539 tablet_bootstrap.cc:492] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: No bootstrap required, opened a new log
I20260809 05:24:19.223783 10539 ts_tablet_manager.cc:1397] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.008s
I20260809 05:24:19.233103 10540 raft_consensus.cc:348] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.233747 10540 raft_consensus.cc:374] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:19.233846 10540 raft_consensus.cc:729] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52327e14a8364090a96b97b8d07f7bd5, State: Initialized, Role: FOLLOWER
I20260809 05:24:19.233588 10541 raft_consensus.cc:348] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.234030 10541 raft_consensus.cc:374] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:19.234144 10541 raft_consensus.cc:729] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50dcbcacb85741a0966aac7200c358bc, State: Initialized, Role: FOLLOWER
I20260809 05:24:19.234637 10539 raft_consensus.cc:348] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.234764 10540 consensus_queue.cc:260] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.234848 10541 consensus_queue.cc:260] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.235074 10539 raft_consensus.cc:374] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:19.235172 10539 raft_consensus.cc:729] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222ab1309f2549d0ab18039d4fa95e15, State: Initialized, Role: FOLLOWER
I20260809 05:24:19.235877 10539 consensus_queue.cc:260] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.239621 10539 ts_tablet_manager.cc:1428] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Time spent starting tablet: real 0.016s user 0.005s sys 0.009s
I20260809 05:24:19.240134 10080 heartbeater.cc:503] Master 127.22.1.126:40971 was elected leader, sending a full tablet report...
I20260809 05:24:19.250612 10540 ts_tablet_manager.cc:1428] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Time spent starting tablet: real 0.028s user 0.009s sys 0.018s
I20260809 05:24:19.251008 10541 ts_tablet_manager.cc:1428] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Time spent starting tablet: real 0.028s user 0.024s sys 0.004s
I20260809 05:24:19.251168 10494 heartbeater.cc:503] Master 127.22.1.126:40971 was elected leader, sending a full tablet report...
W20260809 05:24:19.386974 10292 tablet.cc:2369] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:24:19.418545 10081 tablet.cc:2369] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:24:19.426352 10503 tablet.cc:2369] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:19.545866 10556 raft_consensus.cc:482] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:24:19.546255 10556 raft_consensus.cc:504] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.548205 10556 leader_election.cc:288] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 222ab1309f2549d0ab18039d4fa95e15 (127.22.1.65:43265), 52327e14a8364090a96b97b8d07f7bd5 (127.22.1.67:40815)
I20260809 05:24:19.557237 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.548621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57647 (local address 127.22.1.67:40815)
0809 05:24:19.548822 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.548829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.549526 (+ 697us) server_negotiation.cc:407] Connection header received
0809 05:24:19.549769 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.549776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.549857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.549956 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:19.550728 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.551832 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.554708 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.555964 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.556114 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.556232 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.556795 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.556947 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:24:19.557299 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.548846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:38691 (local address 127.22.1.65:43265)
0809 05:24:19.549070 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.549075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.549638 (+ 563us) server_negotiation.cc:407] Connection header received
0809 05:24:19.549772 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.549776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.549857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.549956 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:19.550728 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.551832 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.554708 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.555968 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.556114 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.556232 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.556795 (+ 563us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.556947 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:24:19.557395 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.548592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43265 (local address 127.22.1.66:38691)
0809 05:24:19.549552 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:19.549567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:19.549673 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:19.549998 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:19.550002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:19.550030 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:19.550558 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.550572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.551999 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.552006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.554491 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.554510 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.556249 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.556257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.556546 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.556609 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 05:24:19.556712 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 05:24:19.556909 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":255,"thread_start_us":390,"threads_started":1}
I20260809 05:24:19.557811 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.548485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:40815 (local address 127.22.1.66:57647)
0809 05:24:19.549367 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:19.549393 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:19.549590 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:19.549998 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:19.550002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:19.550099 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:19.550574 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.550584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.552001 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.552006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.554498 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:19.554510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.556277 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.556282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:19.556555 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.556611 (+ 56us) client_negotiation.cc:769] Sending connection context
0809 05:24:19.556712 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 05:24:19.556909 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":160,"spinlock_wait_cycles":27392,"thread_start_us":748,"threads_started":1}
I20260809 05:24:19.558516 10005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820372e358f43ad9e738364f9100682" candidate_uuid: "50dcbcacb85741a0966aac7200c358bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222ab1309f2549d0ab18039d4fa95e15" is_pre_election: true
I20260809 05:24:19.558642 10427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820372e358f43ad9e738364f9100682" candidate_uuid: "50dcbcacb85741a0966aac7200c358bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52327e14a8364090a96b97b8d07f7bd5" is_pre_election: true
I20260809 05:24:19.559055 10427 raft_consensus.cc:2454] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dcbcacb85741a0966aac7200c358bc in term 0.
I20260809 05:24:19.559046 10005 raft_consensus.cc:2454] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50dcbcacb85741a0966aac7200c358bc in term 0.
I20260809 05:24:19.559636 10136 leader_election.cc:302] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50dcbcacb85741a0966aac7200c358bc, 52327e14a8364090a96b97b8d07f7bd5; no voters:
I20260809 05:24:19.560045 10556 raft_consensus.cc:2781] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:24:19.560176 10556 raft_consensus.cc:482] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:24:19.560274 10556 raft_consensus.cc:3037] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:19.562609 10556 raft_consensus.cc:504] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.563237 10556 leader_election.cc:288] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [CANDIDATE]: Term 1 election: Requested vote from peers 222ab1309f2549d0ab18039d4fa95e15 (127.22.1.65:43265), 52327e14a8364090a96b97b8d07f7bd5 (127.22.1.67:40815)
I20260809 05:24:19.563750 10005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820372e358f43ad9e738364f9100682" candidate_uuid: "50dcbcacb85741a0966aac7200c358bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222ab1309f2549d0ab18039d4fa95e15"
I20260809 05:24:19.563865 10427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6820372e358f43ad9e738364f9100682" candidate_uuid: "50dcbcacb85741a0966aac7200c358bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52327e14a8364090a96b97b8d07f7bd5"
I20260809 05:24:19.563938 10005 raft_consensus.cc:3037] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:19.564097 10427 raft_consensus.cc:3037] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:19.566232 10005 raft_consensus.cc:2454] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dcbcacb85741a0966aac7200c358bc in term 1.
I20260809 05:24:19.566279 10427 raft_consensus.cc:2454] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50dcbcacb85741a0966aac7200c358bc in term 1.
I20260809 05:24:19.566710 10135 leader_election.cc:302] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 222ab1309f2549d0ab18039d4fa95e15, 50dcbcacb85741a0966aac7200c358bc, 52327e14a8364090a96b97b8d07f7bd5; no voters:
I20260809 05:24:19.567019 10556 raft_consensus.cc:2781] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:19.567479 10556 raft_consensus.cc:686] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 1 LEADER]: Becoming Leader. State: Replica: 50dcbcacb85741a0966aac7200c358bc, State: Running, Role: LEADER
I20260809 05:24:19.568002 10556 consensus_queue.cc:237] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } }
I20260809 05:24:19.572642 9773 catalog_manager.cc:5495] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 50dcbcacb85741a0966aac7200c358bc (127.22.1.66). New cstate: current_term: 1 leader_uuid: "50dcbcacb85741a0966aac7200c358bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50dcbcacb85741a0966aac7200c358bc" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 } health_report { overall_health: UNKNOWN } } }
I20260809 05:24:19.613741 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.608203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.22.1.66:46129)
0809 05:24:19.608385 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.608390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.608648 (+ 258us) server_negotiation.cc:407] Connection header received
0809 05:24:19.608853 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.608857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.608945 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.609069 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:24:19.609528 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.610282 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.611684 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.612096 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.612209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.612691 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:24:19.613278 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:24:19.613389 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.613466 (+ 77us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.613535 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 05:24:19.622367 10005 raft_consensus.cc:1261] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 1 FOLLOWER]: Refusing update from remote peer 50dcbcacb85741a0966aac7200c358bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:19.622370 10427 raft_consensus.cc:1261] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 1 FOLLOWER]: Refusing update from remote peer 50dcbcacb85741a0966aac7200c358bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:19.623454 10556 consensus_queue.cc:1035] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "52327e14a8364090a96b97b8d07f7bd5" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 40815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:19.623778 10562 consensus_queue.cc:1035] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "222ab1309f2549d0ab18039d4fa95e15" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:19.649729 9772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:19.649999 9772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:19.653782 9772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 93cc8efd5b0c484c85486fc5d6bf8ff8: Sending DeleteTablet for 3 replicas of tablet 6820372e358f43ad9e738364f9100682
I20260809 05:24:19.655121 10397 tablet_service.cc:1514] Processing DeleteTablet for tablet 6820372e358f43ad9e738364f9100682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:19 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37046
I20260809 05:24:19.655115 10200 tablet_service.cc:1514] Processing DeleteTablet for tablet 6820372e358f43ad9e738364f9100682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:19 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55220
I20260809 05:24:19.655435 9971 tablet_service.cc:1514] Processing DeleteTablet for tablet 6820372e358f43ad9e738364f9100682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:19 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:43334
I20260809 05:24:19.657977 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 9829
I20260809 05:24:19.668227 10589 tablet_replica.cc:330] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: stopping tablet replica
I20260809 05:24:19.669026 10589 raft_consensus.cc:2229] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:19.669824 10589 raft_consensus.cc:2258] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:19.672818 10589 ts_tablet_manager.cc:1905] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:19.681072 10589 ts_tablet_manager.cc:1918] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:19.681434 10589 log.cc:1196] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/6820372e358f43ad9e738364f9100682
I20260809 05:24:19.682106 10589 ts_tablet_manager.cc:1939] T 6820372e358f43ad9e738364f9100682 P 52327e14a8364090a96b97b8d07f7bd5: Deleting consensus metadata
I20260809 05:24:19.684384 9755 catalog_manager.cc:4841] TS 52327e14a8364090a96b97b8d07f7bd5 (127.22.1.67:40815): tablet 6820372e358f43ad9e738364f9100682 (table test-table [id=3dcdd10b44b747b294e261c1b87562ab]) successfully deleted
I20260809 05:24:19.692705 10591 tablet_replica.cc:330] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: stopping tablet replica
I20260809 05:24:19.694202 10591 raft_consensus.cc:2229] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:24:19.695070 10591 raft_consensus.cc:2258] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:19.695657 10592 tablet_replica.cc:330] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: stopping tablet replica
I20260809 05:24:19.696089 10592 raft_consensus.cc:2229] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:19.696619 10592 raft_consensus.cc:2258] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:19.698871 10592 ts_tablet_manager.cc:1905] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:19.699183 10591 ts_tablet_manager.cc:1905] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:19.706037 10592 ts_tablet_manager.cc:1918] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:19.706210 10592 log.cc:1196] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/6820372e358f43ad9e738364f9100682
I20260809 05:24:19.706763 10592 ts_tablet_manager.cc:1939] T 6820372e358f43ad9e738364f9100682 P 222ab1309f2549d0ab18039d4fa95e15: Deleting consensus metadata
I20260809 05:24:19.708705 9754 catalog_manager.cc:4841] TS 222ab1309f2549d0ab18039d4fa95e15 (127.22.1.65:43265): tablet 6820372e358f43ad9e738364f9100682 (table test-table [id=3dcdd10b44b747b294e261c1b87562ab]) successfully deleted
I20260809 05:24:19.710889 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.658864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.22.1.65:43265)
0809 05:24:19.659087 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.659095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.685406 (+ 26311us) server_negotiation.cc:407] Connection header received
0809 05:24:19.685751 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.685757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.685858 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.685967 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:19.688034 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.689299 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.692177 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.692907 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.693035 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.701645 (+ 8610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:19.701664 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:19.701684 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:19.701733 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:19.706591 (+ 4858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.708032 (+ 1441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.708043 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.708047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.708238 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.708767 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.708772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.708776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.709312 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:19.709598 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.710283 (+ 685us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.710529 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:24:19.711599 10070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:19.712852 10591 ts_tablet_manager.cc:1918] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:19.713066 10591 log.cc:1196] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/6820372e358f43ad9e738364f9100682
I20260809 05:24:19.713678 10591 ts_tablet_manager.cc:1939] T 6820372e358f43ad9e738364f9100682 P 50dcbcacb85741a0966aac7200c358bc: Deleting consensus metadata
I20260809 05:24:19.715207 9754 catalog_manager.cc:4841] TS 50dcbcacb85741a0966aac7200c358bc (127.22.1.66:46129): tablet 6820372e358f43ad9e738364f9100682 (table test-table [id=3dcdd10b44b747b294e261c1b87562ab]) successfully deleted
I20260809 05:24:19.805091 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 9829
I20260809 05:24:19.836445 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 10082
I20260809 05:24:19.852617 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:19.837428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.22.1.66:46129)
0809 05:24:19.837593 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:19.837597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:19.837616 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:24:19.837732 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:19.837736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:19.837821 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:19.837904 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:19.839629 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.840411 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.842340 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:19.842836 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:19.842942 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:19.846423 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:19.846438 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:19.846458 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:19.846506 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:19.849687 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.850581 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.850588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.850591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.850741 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:19.851142 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:19.851145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:19.851147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:19.851505 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:19.851776 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:19.852110 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 05:24:19.852296 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:24:19.853315 10285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:19.982082 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 10082
I20260809 05:24:20.003665 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 10295
I20260809 05:24:20.016857 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:20.004452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.22.1.67:40815)
0809 05:24:20.004632 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:20.004638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:20.004692 (+ 54us) server_negotiation.cc:407] Connection header received
0809 05:24:20.004907 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:20.004911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:20.004992 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:20.005092 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:20.006440 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:20.007175 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:20.008515 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:20.008903 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:20.009013 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:20.011512 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:20.011527 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:20.011546 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:20.011579 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:20.014112 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:20.014828 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:20.014835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:20.014838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:20.014981 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:20.015321 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:20.015325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:20.015327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:20.015681 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:20.015980 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:20.016307 (+ 327us) server_negotiation.cc:299] Negotiation successful
0809 05:24:20.016514 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260809 05:24:20.017704 10484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:20.105471 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 10295
I20260809 05:24:20.128345 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 9726
I20260809 05:24:20.129575 9818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:20.240716 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 9726
I20260809 05:24:20.277202 22533 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260809 05:24:20.623481 22533 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260809 05:24:20.623750 22533 mini_ranger.cc:66] Stopping Ranger...
I20260809 05:24:20.969570 22533 mini_ranger.cc:68] Stopped Ranger
2026-08-09 05:24:20.969 UTC [8239] LOG: received smart shutdown request
2026-08-09 05:24:20.972 UTC [8239] LOG: background worker "logical replication launcher" (PID 8251) exited with exit code 1
2026-08-09 05:24:20.973 UTC [8245] LOG: shutting down
2026-08-09 05:24:21.114 UTC [8239] LOG: database system is shut down
2026-08-09T05:24:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53883 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:24:21 dist-test-slave-rmhg krb5kdc[10617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:24:21 dist-test-slave-rmhg krb5kdc[10617](info): set up 2 sockets
Aug 09 05:24:21 dist-test-slave-rmhg krb5kdc[10617](info): commencing operation
krb5kdc: starting...
W20260809 05:24:22.436923 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:24:22 dist-test-slave-rmhg krb5kdc[10617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:24:22.704839 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:45201
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:23.040359 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:23.040673 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:23.040762 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:23.048040 10638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:23.048161 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:23.048220 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:23.048276 10638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:23.048321 10638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:23.058089 10638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:45201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:23.059393 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:23.060648 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:23.072003 10645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:23.072067 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:23.072060 10646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:23.072966 10638 server_base.cc:1034] running on GCE node
I20260809 05:24:23.073787 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:23.075237 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:23.076443 10638 hybrid_clock.cc:648] HybridClock initialized: now 1786253063076383 us; error 53 us; skew 500 ppm
I20260809 05:24:23.076812 10638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:23.083760 10638 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:23.085417 10638 webserver.cc:460] Webserver started at http://127.22.1.126:44643/ using document root <none> and password file <none>
I20260809 05:24:23.086171 10638 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:23.086298 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:23.086603 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:23.089773 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "965775d4632045ecb7287a97a36ee8ec"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "5ea3e5ef7a51a7227046d3c6f51d5cc9"
server_key_iv: "0fafd6014ca0cf9b5546596b6811453a"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.090626 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "965775d4632045ecb7287a97a36ee8ec"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "5ea3e5ef7a51a7227046d3c6f51d5cc9"
server_key_iv: "0fafd6014ca0cf9b5546596b6811453a"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.096197 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 05:24:23.099889 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:23.101418 10638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:23.101593 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "965775d4632045ecb7287a97a36ee8ec"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "5ea3e5ef7a51a7227046d3c6f51d5cc9"
server_key_iv: "0fafd6014ca0cf9b5546596b6811453a"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.101799 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:23.141880 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:23.146485 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:23.146775 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:23.164382 10638 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:45201
I20260809 05:24:23.164481 10734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:45201 every 8 connection(s)
I20260809 05:24:23.165875 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:23.169595 10737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:23.171806 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 10638
I20260809 05:24:23.172043 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:24:23.172845 22533 external_mini_cluster.cc:1428] Setting key 7489cfc5507b8d085a6cf9ecdf3776e3
I20260809 05:24:23.178084 10737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.1.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.125@KRBTEST.COM" created.
I20260809 05:24:23.203297 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.181903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45201 (local address 127.0.0.1:37370)
0809 05:24:23.183225 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.183269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:23.185178 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:23.186055 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:23.186062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:23.186818 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:23.187309 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.187334 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.188850 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.188855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.190544 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.190560 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.191575 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.191585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.191778 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.192494 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:23.192535 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:23.195120 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:23.198981 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.198991 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.199001 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.199375 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.199871 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.199876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.199879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.200050 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.200848 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:23.200861 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:23.201070 (+ 209us) client_negotiation.cc:769] Sending connection context
0809 05:24:23.201343 (+ 273us) client_negotiation.cc:240] Negotiation successful
0809 05:24:23.201951 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":120,"spinlock_wait_cycles":47360,"thread_start_us":774,"threads_started":1}
I20260809 05:24:23.203330 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.181903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.22.1.126:45201)
0809 05:24:23.182792 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:23.182806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:23.183421 (+ 615us) server_negotiation.cc:407] Connection header received
0809 05:24:23.185564 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:23.185570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:23.185793 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:23.185959 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:23.187508 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.188692 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.190712 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.191300 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.191440 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.195386 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:23.195397 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:23.195417 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:23.195461 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:23.198850 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.199529 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.199539 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.199543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.199741 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.200175 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.200179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.200181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.200705 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:23.200918 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:23.201506 (+ 588us) server_negotiation.cc:299] Negotiation successful
0809 05:24:23.201789 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":415,"threads_started":1}
I20260809 05:24:23.206537 10737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:24:23.207618 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.207169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50044)
0809 05:24:23.207352 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.207477 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260809 05:24:23.207852 10665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
I20260809 05:24:23.209496 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.209046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50058)
0809 05:24:23.209267 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.209346 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 05:24:23.209719 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42959: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
I20260809 05:24:23.232863 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42959
--webserver_interface=127.22.1.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:23.259135 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } attempt: 1
I20260809 05:24:23.260315 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.259869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50072)
0809 05:24:23.260075 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.260185 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 05:24:23.261351 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.260909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50086)
0809 05:24:23.261070 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.261185 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 05:24:23.261551 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42959: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
I20260809 05:24:23.329818 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } attempt: 2
I20260809 05:24:23.331180 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.330592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50094)
0809 05:24:23.330865 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.331005 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260809 05:24:23.332324 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.331866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50102)
0809 05:24:23.332056 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.332147 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 05:24:23.332599 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42959: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
I20260809 05:24:23.423846 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } attempt: 3
I20260809 05:24:23.425069 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.424512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50104)
0809 05:24:23.424757 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.424877 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 05:24:23.426220 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.425754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50118)
0809 05:24:23.425925 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.426047 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 05:24:23.426438 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42959: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
W20260809 05:24:23.568158 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:23.568459 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:23.568608 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 05:24:23.569674 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } attempt: 4
I20260809 05:24:23.570907 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.570289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50134)
0809 05:24:23.570632 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.570751 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260809 05:24:23.571924 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.571533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50150)
0809 05:24:23.571685 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.571781 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 05:24:23.572134 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42959: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42959: connect: Connection refused (error 111)
W20260809 05:24:23.576269 10746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:23.576371 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:23.576442 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:23.576519 10746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:23.576572 10746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:23.586510 10746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.1.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:23.587852 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:23.589181 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:23.600353 10751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:23.600399 10754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:23.600417 10752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:23.601791 10746 server_base.cc:1034] running on GCE node
I20260809 05:24:23.602587 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:23.603930 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:23.605129 10746 hybrid_clock.cc:648] HybridClock initialized: now 1786253063605081 us; error 44 us; skew 500 ppm
I20260809 05:24:23.605472 10746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:23.608547 10746 init.cc:377] Logged in from keytab as kudu/127.22.1.125@KRBTEST.COM (short username kudu)
I20260809 05:24:23.610028 10746 webserver.cc:460] Webserver started at http://127.22.1.125:36411/ using document root <none> and password file <none>
I20260809 05:24:23.610704 10746 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:23.610816 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:23.611114 10746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:23.614084 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data/instance:
uuid: "6b23452e5f674de6bc1c91b7acb933db"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "3767a8a2f23502ee79870f97ffc0ec1a"
server_key_iv: "7c22a6e32538e011edb0e197473fa8f1"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.614859 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal/instance:
uuid: "6b23452e5f674de6bc1c91b7acb933db"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "3767a8a2f23502ee79870f97ffc0ec1a"
server_key_iv: "7c22a6e32538e011edb0e197473fa8f1"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.619930 10746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:24:23.623274 10770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:23.624531 10746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:24:23.624687 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "6b23452e5f674de6bc1c91b7acb933db"
format_stamp: "Formatted at 2026-08-09 05:24:23 on dist-test-slave-rmhg"
server_key: "3767a8a2f23502ee79870f97ffc0ec1a"
server_key_iv: "7c22a6e32538e011edb0e197473fa8f1"
server_key_version: "encryptionkey@0"
I20260809 05:24:23.624856 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:23.641645 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:23.645860 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:23.646121 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:23.663163 10746 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.125:42959
I20260809 05:24:23.663200 10846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.125:42959 every 8 connection(s)
I20260809 05:24:23.664526 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
I20260809 05:24:23.667959 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 10746
I20260809 05:24:23.668146 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-1/wal/instance
I20260809 05:24:23.668398 10849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:23.668630 22533 external_mini_cluster.cc:1428] Setting key 1d4d8288d81f28c453ad25bdd5eac630
I20260809 05:24:23.674901 10849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.1.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.124@KRBTEST.COM" created.
I20260809 05:24:23.694783 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.677510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.22.1.126:45201)
0809 05:24:23.677728 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:23.677733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:23.679030 (+ 1297us) server_negotiation.cc:407] Connection header received
0809 05:24:23.680868 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:23.680874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:23.680957 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:23.681054 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:23.682559 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.683226 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.684747 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.685202 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.685316 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.688697 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:23.688704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:23.688708 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:23.688752 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:23.691954 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.692635 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.692639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.692641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.692794 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.693174 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.693178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.693181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.693445 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:23.693560 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:23.694279 (+ 719us) server_negotiation.cc:299] Negotiation successful
0809 05:24:23.694508 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:24:23.695935 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.678054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45201 (local address 127.0.0.1:37372)
0809 05:24:23.678843 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.678890 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:23.680590 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:23.681319 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:23.681327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:23.681828 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:23.682381 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.682397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.683375 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.683379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.684604 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.684617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.685418 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.685426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.685647 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.686277 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:23.686300 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:23.688409 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:23.692088 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.692095 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.692108 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.692542 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.692882 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.692885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.692887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.693072 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.693562 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:23.693568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:23.693914 (+ 346us) client_negotiation.cc:769] Sending connection context
0809 05:24:23.694159 (+ 245us) client_negotiation.cc:240] Negotiation successful
0809 05:24:23.694602 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":367,"threads_started":1}
I20260809 05:24:23.697649 10849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:24:23.711745 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.698426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50164)
0809 05:24:23.698600 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.698622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:23.698707 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:23.699609 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:23.699614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:23.700030 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:23.700156 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.700164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.701514 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.701519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.702322 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.702332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.703092 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.703098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.703274 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.703806 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:23.703822 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:23.705838 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:23.709539 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.709543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.709546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.709878 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.710264 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.710267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.710269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.710404 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.711053 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:23.711056 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:23.711189 (+ 133us) client_negotiation.cc:769] Sending connection context
0809 05:24:23.711292 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 05:24:23.711475 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 05:24:23.712101 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.698636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50164 (local address 127.22.1.125:42959)
0809 05:24:23.699233 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:23.699239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:23.699261 (+ 22us) server_negotiation.cc:407] Connection header received
0809 05:24:23.699300 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:23.699306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:23.699466 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:23.699609 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:23.700300 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.701382 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.702442 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.702959 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.703066 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.706070 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:23.706084 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:23.706105 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:23.706143 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:23.709397 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.710012 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.710021 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.710025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.710162 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.710506 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.710510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.710513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.710926 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:23.711139 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:23.711463 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 05:24:23.711740 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":326,"threads_started":1}
I20260809 05:24:23.714434 10849 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:24:23.715608 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.715166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50240)
0809 05:24:23.715357 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.715468 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260809 05:24:23.715905 10776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:23.717501 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.717211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50254)
0809 05:24:23.717331 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.717404 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 05:24:23.717751 10849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
I20260809 05:24:23.726927 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:43189
--webserver_interface=127.22.1.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:23.767051 10849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 1
I20260809 05:24:23.768229 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.767755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50270)
0809 05:24:23.767945 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.768077 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 05:24:23.769255 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.768875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50274)
0809 05:24:23.769031 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.769110 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 05:24:23.769451 10849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:23.837658 10849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 2
I20260809 05:24:23.838697 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.838250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50276)
0809 05:24:23.838445 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.838539 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 05:24:23.839705 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.839377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50292)
0809 05:24:23.839518 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.839590 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 05:24:23.839907 10849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:23.871403 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } attempt: 5
Aug 09 05:24:23 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:24:23.885576 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.872277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.22.1.125:42959)
0809 05:24:23.872471 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:23.872477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:23.872501 (+ 24us) server_negotiation.cc:407] Connection header received
0809 05:24:23.872615 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:23.872621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:23.872724 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:23.872847 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:23.873681 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.874799 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.875996 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.876674 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.876804 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.880178 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:23.880184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:23.880187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:23.880224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:23.882825 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.883647 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.883651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.883654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.883775 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:23.884158 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:23.884161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:23.884163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:23.884472 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:23.884589 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:23.885107 (+ 518us) server_negotiation.cc:299] Negotiation successful
0809 05:24:23.885340 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 05:24:23.885664 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.872169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50166)
0809 05:24:23.872354 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.872375 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:23.872490 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:23.872865 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:23.872869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:23.873303 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:23.873493 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.873502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.874951 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.874955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.875848 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:23.875862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:23.876858 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:23.876863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:23.877019 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:23.877705 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:23.877723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:23.879956 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:23.883000 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.883006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.883010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.883530 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.883877 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:23.883882 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:23.883885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:23.884056 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:23.884587 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:23.884592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:23.884775 (+ 183us) client_negotiation.cc:769] Sending connection context
0809 05:24:23.884963 (+ 188us) client_negotiation.cc:240] Negotiation successful
0809 05:24:23.885313 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260809 05:24:23.886854 10737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:24:23.888113 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.887645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50308)
0809 05:24:23.887830 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.887926 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260809 05:24:23.889132 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.888732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50322)
0809 05:24:23.888893 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.888991 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 05:24:23.889392 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:23.931113 10849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 3
I20260809 05:24:23.932214 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.931743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50334)
0809 05:24:23.931950 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.932054 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260809 05:24:23.933393 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.932920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50348)
0809 05:24:23.933183 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.933273 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260809 05:24:23.933619 10849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:23.940641 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 1
I20260809 05:24:23.941852 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.941331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50356)
0809 05:24:23.941540 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.941652 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 05:24:23.942890 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:23.942512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50360)
0809 05:24:23.942657 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:23.942764 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 05:24:23.943116 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:24.011343 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 2
I20260809 05:24:24.012614 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.012046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50376)
0809 05:24:24.012284 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.012399 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 05:24:24.013633 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.013272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50392)
0809 05:24:24.013411 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.013499 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 05:24:24.013878 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
W20260809 05:24:24.074335 10857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:24.074658 10857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:24.074751 10857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260809 05:24:24.076848 10849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 4
I20260809 05:24:24.078126 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.077609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50406)
0809 05:24:24.077862 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.077959 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260809 05:24:24.079113 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.078791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50422)
0809 05:24:24.078913 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.078986 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 05:24:24.079312 10849 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
W20260809 05:24:24.082814 10857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:24.082927 10857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:24.082999 10857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:24.083070 10857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:24.083128 10857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:24.093501 10857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:45201,127.22.1.125:42959,127.22.1.124:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.1.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:24.094923 10857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:24.096256 10857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:24.108184 10862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:24.108161 10863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:24.108749 10866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:24.108799 10857 server_base.cc:1034] running on GCE node
I20260809 05:24:24.109685 10857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:24.111228 10857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:24.112438 10857 hybrid_clock.cc:648] HybridClock initialized: now 1786253064112393 us; error 37 us; skew 500 ppm
I20260809 05:24:24.112811 10857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253064, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:24.116088 10857 init.cc:377] Logged in from keytab as kudu/127.22.1.124@KRBTEST.COM (short username kudu)
I20260809 05:24:24.117832 10857 webserver.cc:460] Webserver started at http://127.22.1.124:35741/ using document root <none> and password file <none>
I20260809 05:24:24.118629 10857 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:24.118755 10857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:24.119067 10857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:24.120126 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 3
I20260809 05:24:24.121421 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.120798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50432)
0809 05:24:24.121052 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.121145 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260809 05:24:24.122413 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.122047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50446)
0809 05:24:24.122234 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.122306 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260809 05:24:24.122817 10737 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:43189: Network error: Client connection negotiation failed: client connection to 127.22.1.124:43189: connect: Connection refused (error 111)
I20260809 05:24:24.122778 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data/instance:
uuid: "836ccde4d8034e06ace5aebd46cd4652"
format_stamp: "Formatted at 2026-08-09 05:24:24 on dist-test-slave-rmhg"
server_key: "cc8a19b29ec618b7ce4899e93a21c60d"
server_key_iv: "2e8f3b980c943d00af2935696f452fd2"
server_key_version: "encryptionkey@0"
I20260809 05:24:24.123637 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal/instance:
uuid: "836ccde4d8034e06ace5aebd46cd4652"
format_stamp: "Formatted at 2026-08-09 05:24:24 on dist-test-slave-rmhg"
server_key: "cc8a19b29ec618b7ce4899e93a21c60d"
server_key_iv: "2e8f3b980c943d00af2935696f452fd2"
server_key_version: "encryptionkey@0"
I20260809 05:24:24.129140 10857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:24:24.133049 10874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:24.134670 10857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:24:24.134864 10857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "836ccde4d8034e06ace5aebd46cd4652"
format_stamp: "Formatted at 2026-08-09 05:24:24 on dist-test-slave-rmhg"
server_key: "cc8a19b29ec618b7ce4899e93a21c60d"
server_key_iv: "2e8f3b980c943d00af2935696f452fd2"
server_key_version: "encryptionkey@0"
I20260809 05:24:24.135079 10857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:24.157409 10857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:24.161633 10857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:24.161909 10857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:24.181099 10857 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.124:43189
I20260809 05:24:24.181130 10950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.124:43189 every 8 connection(s)
I20260809 05:24:24.182533 10857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
I20260809 05:24:24.183046 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 10857
I20260809 05:24:24.183220 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786252853919692-22533-0/minicluster-data/master-2/wal/instance
I20260809 05:24:24.183877 22533 external_mini_cluster.cc:1428] Setting key e6a03398b4ec329de462b3c3100bec27
I20260809 05:24:24.187057 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:24.194186 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253064, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:24.207716 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.185941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50448 (local address 127.22.1.124:43189)
0809 05:24:24.186756 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.186770 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.186812 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:24:24.188538 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.188564 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.188978 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.189416 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.190944 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.192340 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.193849 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.194581 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.194791 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.199417 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.199425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.199436 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.199472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.202751 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.203513 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.203521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.203528 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.203722 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.204199 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.204203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.204206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.204867 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.205128 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.205805 (+ 677us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.206328 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260809 05:24:24.212718 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.196894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:45201 (local address 127.0.0.1:37388)
0809 05:24:24.197688 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.197741 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:24.197905 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:24.198725 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:24.198733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:24.199515 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:24.199844 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.199854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.201350 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.201357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.202550 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.202564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.203434 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.203441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.203615 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.204180 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:24.204198 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:24.206613 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:24.210097 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.210105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.210110 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.210573 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.210968 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.210973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.210976 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.211176 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.211769 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:24.211777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:24.211990 (+ 213us) client_negotiation.cc:769] Sending connection context
0809 05:24:24.212187 (+ 197us) client_negotiation.cc:240] Negotiation successful
0809 05:24:24.212425 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":327,"threads_started":1}
I20260809 05:24:24.212992 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.197020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.22.1.126:45201)
0809 05:24:24.198376 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.198380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.198396 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:24:24.198446 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.198449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.198505 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.198600 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.200031 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.201124 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.202720 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.203251 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.203367 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.206840 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.206847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.206852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.206896 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.209953 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.210690 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.210696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.210700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.210850 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.211312 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.211318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.211321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.211639 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.211750 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.212343 (+ 593us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.212585 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":1126,"threads_started":1}
I20260809 05:24:24.214432 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253064, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:24:24.228147 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.211051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.22.1.126:45201)
0809 05:24:24.211595 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.211602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.211622 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:24:24.211662 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.211666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.211730 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.211837 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.213483 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.214586 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.216335 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.216868 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.216986 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.221922 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.221928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.221932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.221976 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.225353 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.226119 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.226124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.226129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.226288 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.226649 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.226654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.226657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.226969 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.227172 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.227524 (+ 352us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.227733 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":253,"threads_started":1}
I20260809 05:24:24.229503 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.215196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.22.1.125:42959)
0809 05:24:24.215347 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.215351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.215558 (+ 207us) server_negotiation.cc:407] Connection header received
0809 05:24:24.215689 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.215693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.215752 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.215826 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.216631 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.217556 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.219303 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.219729 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.219822 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.223444 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.223451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.223456 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.223498 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.226564 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.227289 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.227293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.227295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.227443 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.227912 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.227917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.227921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.228252 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.228439 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.228894 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.229109 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:24:24.229516 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.215222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42959 (local address 127.0.0.1:50174)
0809 05:24:24.215464 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.215481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:24.215591 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:24.215866 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:24.215871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:24.216328 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:24.216490 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.216499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.217765 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.217772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.219154 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.219170 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.219885 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.219891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.220108 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.220669 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:24.220692 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:24.223234 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:24.226710 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.226714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.226716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.227161 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.227588 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.227592 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.227596 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.227800 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.228495 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:24.228498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:24.228627 (+ 129us) client_negotiation.cc:769] Sending connection context
0809 05:24:24.228787 (+ 160us) client_negotiation.cc:240] Negotiation successful
0809 05:24:24.229130 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 05:24:24.230940 10953 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253064, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:24:24.245054 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.231814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.22.1.124:43189)
0809 05:24:24.231992 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.231997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.232175 (+ 178us) server_negotiation.cc:407] Connection header received
0809 05:24:24.232301 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.232304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.232358 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.232431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.233248 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.234315 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.235832 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.236408 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.236545 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.239614 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.239618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.239622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.239651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.242277 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.243059 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.243064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.243066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.243189 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.243550 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.243553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.243555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.243838 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.243966 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.244327 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.244603 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":57}
I20260809 05:24:24.245172 10958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.231816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50456)
0809 05:24:24.232094 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.232112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:24.232212 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:24.232470 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:24.232474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:24.232929 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:24.233097 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.233108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.234458 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.234462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.235695 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.235707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.236591 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.236596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.236726 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.237267 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:24.237282 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:24.239395 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:24.242428 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.242434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.242439 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.242916 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.243299 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.243304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.243308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.243440 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.243947 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:24.243952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:24.244094 (+ 142us) client_negotiation.cc:769] Sending connection context
0809 05:24:24.244229 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 05:24:24.244765 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":179}
I20260809 05:24:24.245728 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.230139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.22.1.125:42959)
0809 05:24:24.230274 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.230278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.230294 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:24:24.230331 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.230334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.230383 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.230458 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.231968 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.233043 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.234754 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.235391 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.235516 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.240229 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.240234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.240237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.240269 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.243088 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.243785 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.243791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.243794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.243939 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.244302 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.244306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.244308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.244588 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.244852 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.245186 (+ 334us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.245403 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260809 05:24:24.250566 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:24:24.251104 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: Bootstrap starting.
I20260809 05:24:24.254406 10953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:24.255915 10953 log.cc:824] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:24.262521 10953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: No bootstrap required, opened a new log
I20260809 05:24:24.273716 10953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.274209 10953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:24.274335 10953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836ccde4d8034e06ace5aebd46cd4652, State: Initialized, Role: FOLLOWER
I20260809 05:24:24.275226 10953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.276322 10976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.276762 10976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.277789 10953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:24.277997 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.255615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.22.1.125:42959)
0809 05:24:24.256225 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.256230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.259566 (+ 3336us) server_negotiation.cc:407] Connection header received
0809 05:24:24.259921 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.259927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.260008 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.260094 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.262328 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.263421 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.265686 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.266336 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.266432 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.270819 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.270826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.270831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.270874 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.274287 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.275180 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.275186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.275190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.275361 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.275877 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.275883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.275886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.276296 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.276527 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.277383 (+ 856us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.277634 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490}
I20260809 05:24:24.280741 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.255338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.22.1.126:45201)
0809 05:24:24.255528 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.255533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.263904 (+ 8371us) server_negotiation.cc:407] Connection header received
0809 05:24:24.264500 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.264506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.264615 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.264689 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.267144 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.268159 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.270418 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.271004 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.271095 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.274134 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.274141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.274145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.274189 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.277376 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.278272 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.278277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.278281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.278432 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.278875 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.278880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.278882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.279258 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.279497 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.280120 (+ 623us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.280385 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 05:24:24.282218 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.255200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.22.1.124:43189)
0809 05:24:24.255403 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.255409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.259798 (+ 4389us) server_negotiation.cc:407] Connection header received
0809 05:24:24.259997 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.260003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.260080 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.260180 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.262703 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.263821 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.266112 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.266823 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.266921 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.270800 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.270811 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.270815 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.270860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.278887 (+ 8027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.279744 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.279749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.279753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.279944 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.280420 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.280425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.280428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.280738 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.280984 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.281796 (+ 812us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.281979 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260809 05:24:24.297259 10995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:24:24.297438 10995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:24:24.300024 10737 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 4
I20260809 05:24:24.290810 10857 master_runner.cc:186] Error getting master registration for 127.22.1.126:45201: OK, instance_id { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" instance_seqno: 1786253063159327 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:24:24.301028 10857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:24:24.315225 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.300726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.22.1.124:43189)
0809 05:24:24.300888 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.300894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.300917 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:24:24.301122 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.301127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.301276 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.301377 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.302260 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.303460 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.304894 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.305409 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.305533 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.308930 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.308936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.308939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.308982 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.312389 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.313216 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.313220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.313226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.313367 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.313740 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.313744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.313749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.314047 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.314175 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.314630 (+ 455us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.314857 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:24:24.315244 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.300654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50474)
0809 05:24:24.300851 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.300872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:24.300999 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:24.301456 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:24.301459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:24.301910 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:24.302085 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.302094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.303627 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.303633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.304722 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.304737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.305570 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.305576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.305749 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.306350 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:24.306368 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:24.308631 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:24.312530 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.312536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.312541 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.313055 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.313490 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.313494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.313496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.313634 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.314182 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:24.314188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:24.314366 (+ 178us) client_negotiation.cc:769] Sending connection context
0809 05:24:24.314504 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 05:24:24.314846 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 05:24:24.322428 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: Bootstrap starting.
I20260809 05:24:24.326915 10737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:24.328379 10737 log.cc:824] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:24.333042 10737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: No bootstrap required, opened a new log
I20260809 05:24:24.344308 10737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.344774 10737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:24.344902 10737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 965775d4632045ecb7287a97a36ee8ec, State: Initialized, Role: FOLLOWER
I20260809 05:24:24.345640 10737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.346567 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.346889 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.348095 10737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: configured and running, proceeding with master startup.
W20260809 05:24:24.357844 11013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:24:24.359288 11013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:24:24.359081 10638 master_runner.cc:186] Error getting master registration for 127.22.1.125:42959: OK, instance_id { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" instance_seqno: 1786253063658368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:24:24.370024 10638 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 05:24:24.378540 10849 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } attempt: 5
Aug 09 05:24:24 dist-test-slave-rmhg krb5kdc[10617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253063, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:24:24.398674 10853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.379370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:43189 (local address 127.0.0.1:50482)
0809 05:24:24.379659 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:24.379680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:24.379806 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:24.380425 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:24.380431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:24.380985 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:24.381231 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.381245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.382779 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.382785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.384209 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:24.384227 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.385288 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.385295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:24.385514 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.388348 (+ 2834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:24.388374 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:24.391313 (+ 2939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:24.395637 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.395644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.395649 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.396225 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.396717 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:24.396723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:24.396727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:24.396918 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:24.397601 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:24.397606 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:24.397778 (+ 172us) client_negotiation.cc:769] Sending connection context
0809 05:24:24.397949 (+ 171us) client_negotiation.cc:240] Negotiation successful
0809 05:24:24.398270 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260809 05:24:24.398797 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:24.379522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.22.1.124:43189)
0809 05:24:24.379694 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:24.379699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:24.379916 (+ 217us) server_negotiation.cc:407] Connection header received
0809 05:24:24.379983 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:24.379988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:24.380083 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:24.380178 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:24.381417 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.382600 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.384398 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:24.385101 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:24.385274 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:24.391545 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:24.391553 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:24.391559 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:24.391604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:24.395467 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.396401 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.396407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.396411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.396590 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:24.397048 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:24.397054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:24.397057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:24.397472 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:24.397636 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:24.398085 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 05:24:24.398362 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260809 05:24:24.399806 10670 tablet.cc:2369] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:24.407106 10849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: Bootstrap starting.
I20260809 05:24:24.411429 10849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:24.412793 10849 log.cc:824] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: Log is configured to *not* fsync() on all Append() calls
W20260809 05:24:24.414194 10881 tablet.cc:2369] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:24.416643 10849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: No bootstrap required, opened a new log
I20260809 05:24:24.427896 10849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.428354 10849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:24.428476 10849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b23452e5f674de6bc1c91b7acb933db, State: Initialized, Role: FOLLOWER
I20260809 05:24:24.429204 10849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.430233 11016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.430789 11016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.431579 10849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:24.453826 10746 master_runner.cc:263] No leader master found from master 6b23452e5f674de6bc1c91b7acb933db
I20260809 05:24:24.454006 10746 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260809 05:24:24.455240 11032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:24:24.455389 11032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:24:24.612689 10976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:24:24.613057 10976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.616039 10704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "836ccde4d8034e06ace5aebd46cd4652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965775d4632045ecb7287a97a36ee8ec" is_pre_election: true
I20260809 05:24:24.616707 10704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836ccde4d8034e06ace5aebd46cd4652 in term 0.
I20260809 05:24:24.617580 10876 leader_election.cc:302] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 836ccde4d8034e06ace5aebd46cd4652, 965775d4632045ecb7287a97a36ee8ec; no voters:
I20260809 05:24:24.618352 10976 leader_election.cc:288] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 965775d4632045ecb7287a97a36ee8ec (127.22.1.126:45201), 6b23452e5f674de6bc1c91b7acb933db (127.22.1.125:42959)
I20260809 05:24:24.618577 10976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:24:24.618723 10976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:24:24.619935 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "836ccde4d8034e06ace5aebd46cd4652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b23452e5f674de6bc1c91b7acb933db" is_pre_election: true
I20260809 05:24:24.620599 10812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 836ccde4d8034e06ace5aebd46cd4652 in term 0.
I20260809 05:24:24.618811 10976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:24.624799 10976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
I20260809 05:24:24.629575 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "836ccde4d8034e06ace5aebd46cd4652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b23452e5f674de6bc1c91b7acb933db"
I20260809 05:24:24.629849 10812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:24.634215 10812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836ccde4d8034e06ace5aebd46cd4652 in term 1.
I20260809 05:24:24.634935 10877 leader_election.cc:302] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b23452e5f674de6bc1c91b7acb933db, 836ccde4d8034e06ace5aebd46cd4652; no voters:
I20260809 05:24:24.635627 10976 leader_election.cc:288] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [CANDIDATE]: Term 1 election: Requested vote from peers 965775d4632045ecb7287a97a36ee8ec (127.22.1.126:45201), 6b23452e5f674de6bc1c91b7acb933db (127.22.1.125:42959)
I20260809 05:24:24.636103 10976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:24.641496 10976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [term 1 LEADER]: Becoming Leader. State: Replica: 836ccde4d8034e06ace5aebd46cd4652, State: Running, Role: LEADER
I20260809 05:24:24.642568 10976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } }
W20260809 05:24:24.648872 10781 tablet.cc:2369] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:24.653465 10704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "836ccde4d8034e06ace5aebd46cd4652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "965775d4632045ecb7287a97a36ee8ec"
I20260809 05:24:24.653757 10704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:24.657583 10704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 836ccde4d8034e06ace5aebd46cd4652 in term 1.
I20260809 05:24:24.664135 11035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836ccde4d8034e06ace5aebd46cd4652. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.664752 11035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:24.672567 11037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:24.675344 11037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:24.697172 10812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [term 1 FOLLOWER]: Refusing update from remote peer 836ccde4d8034e06ace5aebd46cd4652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:24.698421 11035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:24.706447 10704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [term 1 FOLLOWER]: Refusing update from remote peer 836ccde4d8034e06ace5aebd46cd4652: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:24.707888 11035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:24.750540 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836ccde4d8034e06ace5aebd46cd4652. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.750880 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.753499 11039 mvcc.cc:205] Tried to move back new op lower bound from 7316492552985182208 to 7316492552788844544. Current Snapshot: MvccSnapshot[applied={T|T < 7316492552985182208}]
I20260809 05:24:24.758785 11041 mvcc.cc:205] Tried to move back new op lower bound from 7316492552985182208 to 7316492552788844544. Current Snapshot: MvccSnapshot[applied={T|T < 7316492552985182208}]
I20260809 05:24:24.764249 11016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 836ccde4d8034e06ace5aebd46cd4652. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.764667 11016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.768772 10976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.769129 10976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:24.774698 11035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.775393 11040 mvcc.cc:205] Tried to move back new op lower bound from 7316492552985182208 to 7316492552788844544. Current Snapshot: MvccSnapshot[applied={T|T < 7316492552985182208}]
I20260809 05:24:24.776007 11035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:24.776551 11016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.776796 11016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.778014 11037 catalog_manager.cc:1347] Generated new cluster ID: 8065d685cd9646ae8f7b37b4135a6418
I20260809 05:24:24.778116 11037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:24.791316 10998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "836ccde4d8034e06ace5aebd46cd4652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "965775d4632045ecb7287a97a36ee8ec" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 45201 } } peers { permanent_uuid: "6b23452e5f674de6bc1c91b7acb933db" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42959 } } peers { permanent_uuid: "836ccde4d8034e06ace5aebd46cd4652" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 43189 } } }
I20260809 05:24:24.791668 10998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:24:24.855368 11037 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:24:24.856797 11037 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:24.868348 11037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 836ccde4d8034e06ace5aebd46cd4652: Generated new TSK 0
I20260809 05:24:24.869668 11037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:24:25.361435 11013 catalog_manager.cc:1259] Loaded cluster ID: 8065d685cd9646ae8f7b37b4135a6418
I20260809 05:24:25.361579 11013 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: loading cluster ID for follower catalog manager: success
I20260809 05:24:25.364059 11013 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: acquiring CA information for follower catalog manager: success
I20260809 05:24:25.366034 11013 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 965775d4632045ecb7287a97a36ee8ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:24:25.457504 11032 catalog_manager.cc:1259] Loaded cluster ID: 8065d685cd9646ae8f7b37b4135a6418
I20260809 05:24:25.457640 11032 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: loading cluster ID for follower catalog manager: success
I20260809 05:24:25.460121 11032 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: acquiring CA information for follower catalog manager: success
I20260809 05:24:25.462060 11032 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b23452e5f674de6bc1c91b7acb933db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:24:25.964378 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 10638
I20260809 05:24:25.966113 10728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:26.075403 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 10638
I20260809 05:24:26.102152 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 10746
I20260809 05:24:26.103317 10838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:26.209789 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 10746
W20260809 05:24:26.236109 10877 connection.cc:535] client connection to 127.22.1.125:42959 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 05:24:26.236416 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 10857
W20260809 05:24:26.236500 10877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 05:24:26.237763 10945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:26.347563 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 10857
2026-08-09T05:24:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5128 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:24:26 dist-test-slave-rmhg krb5kdc[11063](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:24:26 dist-test-slave-rmhg krb5kdc[11063](info): set up 2 sockets
Aug 09 05:24:26 dist-test-slave-rmhg krb5kdc[11063](info): commencing operation
krb5kdc: starting...
W20260809 05:24:27.548856 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.154s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:24:27 dist-test-slave-rmhg krb5kdc[11063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:24:27.822387 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41525
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:28.158633 11080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:28.158977 11080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:28.159101 11080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:28.166494 11080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:28.166644 11080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:28.166716 11080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:28.166792 11080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:28.166854 11080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:28.177022 11080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:28.178514 11080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:28.179904 11080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:28.191994 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:28.192085 11088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:28.192083 11090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:28.193126 11080 server_base.cc:1034] running on GCE node
I20260809 05:24:28.193974 11080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:28.195538 11080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:28.196758 11080 hybrid_clock.cc:648] HybridClock initialized: now 1786253068196697 us; error 43 us; skew 500 ppm
I20260809 05:24:28.197125 11080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:28 dist-test-slave-rmhg krb5kdc[11063](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253068, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:28.203367 11080 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:28.204969 11080 webserver.cc:460] Webserver started at http://127.22.1.126:35557/ using document root <none> and password file <none>
I20260809 05:24:28.205726 11080 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:28.205837 11080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:28.206130 11080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:28.209291 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "6133d447e52145e7b34876df4d975cdc"
format_stamp: "Formatted at 2026-08-09 05:24:28 on dist-test-slave-rmhg"
server_key: "ac0bfdacd7247ac2342cd81dddd9a8d9"
server_key_iv: "78a04b9170712a5d110ecf6bd44c5b35"
server_key_version: "encryptionkey@0"
I20260809 05:24:28.210063 11080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "6133d447e52145e7b34876df4d975cdc"
format_stamp: "Formatted at 2026-08-09 05:24:28 on dist-test-slave-rmhg"
server_key: "ac0bfdacd7247ac2342cd81dddd9a8d9"
server_key_iv: "78a04b9170712a5d110ecf6bd44c5b35"
server_key_version: "encryptionkey@0"
I20260809 05:24:28.215561 11080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:24:28.219297 11103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:28.220778 11080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:24:28.220963 11080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "6133d447e52145e7b34876df4d975cdc"
format_stamp: "Formatted at 2026-08-09 05:24:28 on dist-test-slave-rmhg"
server_key: "ac0bfdacd7247ac2342cd81dddd9a8d9"
server_key_iv: "78a04b9170712a5d110ecf6bd44c5b35"
server_key_version: "encryptionkey@0"
I20260809 05:24:28.221180 11080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:28.240140 11080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:28.244336 11080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:28.244621 11080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:28.262382 11080 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41525
I20260809 05:24:28.262463 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41525 every 8 connection(s)
I20260809 05:24:28.263744 11080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:28.267535 11182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:28.269629 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 11080
I20260809 05:24:28.269822 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:24:28.270481 22533 external_mini_cluster.cc:1428] Setting key 8621d786fd0e50e81e06f237f7f382f3
I20260809 05:24:28.279145 11182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc: Bootstrap starting.
I20260809 05:24:28.282775 11182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:28.284063 11182 log.cc:824] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:24:28 dist-test-slave-rmhg krb5kdc[11063](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:28.286937 11182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc: No bootstrap required, opened a new log
I20260809 05:24:28.293470 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:28.272256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.22.1.126:41525)
0809 05:24:28.273096 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:28.273108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:28.273175 (+ 67us) server_negotiation.cc:407] Connection header received
0809 05:24:28.274860 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:28.274883 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:28.275266 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:28.275598 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:28.277489 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:28.278750 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:28.280584 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:28.281298 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:28.281430 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:28.286336 (+ 4906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:28.286344 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:28.286358 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:28.286393 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:28.289390 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:28.290111 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:28.290121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:28.290127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:28.290300 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:28.290706 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:28.290710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:28.290712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:28.291259 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:28.291543 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:28.291999 (+ 456us) server_negotiation.cc:299] Negotiation successful
0809 05:24:28.292411 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":350,"threads_started":1}
I20260809 05:24:28.294663 11182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } }
I20260809 05:24:28.295051 11182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:28.295149 11182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6133d447e52145e7b34876df4d975cdc, State: Initialized, Role: FOLLOWER
I20260809 05:24:28.295668 11182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } }
I20260809 05:24:28.295936 11182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:28.296077 11182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:28.296216 11182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:28.297364 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:24:28.298610 11182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } }
I20260809 05:24:28.299067 11182 leader_election.cc:302] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6133d447e52145e7b34876df4d975cdc; no voters:
I20260809 05:24:28.299679 11182 leader_election.cc:288] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:24:28.299786 11190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:28.300848 11190 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [term 1 LEADER]: Becoming Leader. State: Replica: 6133d447e52145e7b34876df4d975cdc, State: Running, Role: LEADER
I20260809 05:24:28.301484 11190 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } }
I20260809 05:24:28.301985 11182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:28.304307 11193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6133d447e52145e7b34876df4d975cdc. Latest consensus state: current_term: 1 leader_uuid: "6133d447e52145e7b34876df4d975cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } } }
I20260809 05:24:28.304798 11193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:28.304371 11191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6133d447e52145e7b34876df4d975cdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6133d447e52145e7b34876df4d975cdc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41525 } } }
I20260809 05:24:28.305176 11191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:28.310263 11200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:28.314962 11200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:28.326934 11200 catalog_manager.cc:1347] Generated new cluster ID: 6fbbe96e20104137a8779be08b489a02
I20260809 05:24:28.327080 11200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:28.332044 11215 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 05:24:28.344503 11200 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:24:28.345898 11200 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:28.354120 11200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6133d447e52145e7b34876df4d975cdc: Generated new TSK 0
I20260809 05:24:28.354990 11200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:24:28.410439 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 11080
I20260809 05:24:28.411756 11174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:28.506588 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 11080
2026-08-09T05:24:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2153 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:24:28 dist-test-slave-rmhg krb5kdc[11224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:24:28 dist-test-slave-rmhg krb5kdc[11224](info): set up 2 sockets
Aug 09 05:24:28 dist-test-slave-rmhg krb5kdc[11224](info): commencing operation
krb5kdc: starting...
W20260809 05:24:29.713797 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.006s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:24:29 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:24:29.984470 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:30.321313 11240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:30.321622 11240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:30.321715 11240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:30.329010 11240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:30.329140 11240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:30.329228 11240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:30.329293 11240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:30.329337 11240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:30.339457 11240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:30.340879 11240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:30.342248 11240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:30.354326 11247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:30.354351 11250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:30.354408 11246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:30.355214 11240 server_base.cc:1034] running on GCE node
I20260809 05:24:30.356014 11240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:30.357589 11240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:30.358800 11240 hybrid_clock.cc:648] HybridClock initialized: now 1786253070358752 us; error 35 us; skew 500 ppm
I20260809 05:24:30.359158 11240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:30 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253070, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:30.362190 11240 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:30.363692 11240 webserver.cc:460] Webserver started at http://127.22.1.126:44691/ using document root <none> and password file <none>
I20260809 05:24:30.364385 11240 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:30.364516 11240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:30.364820 11240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:30.367915 11240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "3c8bdf3527a243828957d46269780406"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "1262517ee6b3cd91889563cd9621f700"
server_key_iv: "cd46f5013f8c40430de9994b3130ec34"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.368685 11240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "3c8bdf3527a243828957d46269780406"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "1262517ee6b3cd91889563cd9621f700"
server_key_iv: "cd46f5013f8c40430de9994b3130ec34"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.373826 11240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:24:30.377552 11259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:30.379346 11240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:24:30.379565 11240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "3c8bdf3527a243828957d46269780406"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "1262517ee6b3cd91889563cd9621f700"
server_key_iv: "cd46f5013f8c40430de9994b3130ec34"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.379859 11240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:30.393395 11240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:30.397387 11240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:30.397658 11240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:30.415690 11240 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35749
I20260809 05:24:30.415709 11344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35749 every 8 connection(s)
I20260809 05:24:30.417223 11240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:30.420960 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 11240
I20260809 05:24:30.421271 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:24:30.421316 11347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:30.422181 22533 external_mini_cluster.cc:1428] Setting key 38487b54cc99e7bba2bf49e7bc0bdd2a
I20260809 05:24:30.435252 11347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap starting.
Aug 09 05:24:30 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:30.439137 11347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:30.440260 11347 log.cc:824] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:30.443547 11347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: No bootstrap required, opened a new log
I20260809 05:24:30.446620 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:30.424297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.22.1.126:35749)
0809 05:24:30.425230 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:30.425246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:30.425293 (+ 47us) server_negotiation.cc:407] Connection header received
0809 05:24:30.427567 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:30.427595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:30.428013 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:30.428352 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:30.430188 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:30.431559 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:30.433398 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:30.434098 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:30.434238 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:30.438994 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:30.439002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:30.439022 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:30.439060 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:30.442363 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:30.443208 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:30.443215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:30.443222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:30.443402 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:30.443823 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:30.443827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:30.443829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:30.444389 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:30.444661 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:30.445096 (+ 435us) server_negotiation.cc:299] Negotiation successful
0809 05:24:30.445501 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":406,"threads_started":1}
I20260809 05:24:30.451586 11347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:30.451943 11347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:30.452040 11347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8bdf3527a243828957d46269780406, State: Initialized, Role: FOLLOWER
I20260809 05:24:30.452584 11347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:30.452795 11347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:30.452924 11347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:30.453071 11347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:30.455212 11347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:30.455687 11347 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8bdf3527a243828957d46269780406; no voters:
I20260809 05:24:30.456260 11347 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:24:30.456382 11352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:30.457479 11352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 LEADER]: Becoming Leader. State: Replica: 3c8bdf3527a243828957d46269780406, State: Running, Role: LEADER
I20260809 05:24:30.458118 11352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:30.458488 11347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:30.461179 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8bdf3527a243828957d46269780406. Latest consensus state: current_term: 1 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:30.461581 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:30.461225 11355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:30.462069 11355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:30.463320 11365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:30.468326 11365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:30.477548 11365 catalog_manager.cc:1347] Generated new cluster ID: a9905725d5134e36b39b230db67b8f1e
I20260809 05:24:30.477703 11365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:30.509079 11365 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:24:30.510464 11365 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:30.520252 11365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Generated new TSK 0
I20260809 05:24:30.521136 11365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:24:30.589357 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:30.933285 11388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:30.933607 11388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:30.933754 11388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:30.941174 11388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:30.941314 11388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:30.941480 11388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:30.952039 11388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:30.953414 11388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:30.955070 11388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:30.968890 11395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:30.968839 11398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:30.968793 11396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:30.969794 11388 server_base.cc:1034] running on GCE node
I20260809 05:24:30.970470 11388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:30.971488 11388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:30.972705 11388 hybrid_clock.cc:648] HybridClock initialized: now 1786253070972647 us; error 53 us; skew 500 ppm
I20260809 05:24:30.973019 11388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:30 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253070, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:30.976176 11388 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:30.977736 11388 webserver.cc:460] Webserver started at http://127.22.1.65:37299/ using document root <none> and password file <none>
I20260809 05:24:30.978425 11388 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:30.978555 11388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:30.978852 11388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:30.981894 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "a9260b59fdb9487b8d39b5605082af51"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "4835b7ceb2ec78124869af9eb30b4b08"
server_key_iv: "e9e54f3d1432cd56839b6cc7914531ef"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.982682 11388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "a9260b59fdb9487b8d39b5605082af51"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "4835b7ceb2ec78124869af9eb30b4b08"
server_key_iv: "e9e54f3d1432cd56839b6cc7914531ef"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.988004 11388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:24:30.991536 11409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:30.992874 11388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:24:30.993037 11388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "a9260b59fdb9487b8d39b5605082af51"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "4835b7ceb2ec78124869af9eb30b4b08"
server_key_iv: "e9e54f3d1432cd56839b6cc7914531ef"
server_key_version: "encryptionkey@0"
I20260809 05:24:30.993252 11388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:31.021914 11388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:31.026050 11388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:31.026340 11388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:31.027304 11388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:31.028736 11388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:31.028816 11388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:31.028934 11388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:31.028997 11388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:31.061058 11388 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:33349
I20260809 05:24:31.061132 11564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:33349 every 8 connection(s)
I20260809 05:24:31.062503 11388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:31.068712 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 11388
I20260809 05:24:31.068926 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:24:31.069666 22533 external_mini_cluster.cc:1428] Setting key 621f9de498c65238624385b499216122
Aug 09 05:24:31 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253070, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:31.083411 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:31.065134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:49415 (local address 127.22.1.126:35749)
0809 05:24:31.065678 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:31.065684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:31.066740 (+ 1056us) server_negotiation.cc:407] Connection header received
0809 05:24:31.068803 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:31.068809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:31.068899 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:31.069016 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:31.070809 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.071702 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.073384 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.073797 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.073918 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:31.077596 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:31.077601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:31.077604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:31.077639 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:31.080184 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:31.080975 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:31.080980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:31.080982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:31.081115 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:31.081578 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:31.081582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:31.081584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:31.081858 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:31.082022 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:31.082755 (+ 733us) server_negotiation.cc:299] Negotiation successful
0809 05:24:31.083010 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":268,"threads_started":1}
I20260809 05:24:31.084465 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:31.065664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:49415)
0809 05:24:31.066552 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:31.066609 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:31.068515 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:31.069310 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:31.069320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:31.069901 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:31.070632 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:31.070653 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.071875 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.071879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:31.073226 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:31.073240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.074036 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.074044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:31.074291 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:31.074987 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:31.075012 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:31.077347 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:31.080350 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:31.080360 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:31.080377 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:31.080824 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:31.081241 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:31.081246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:31.081250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:31.081455 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:31.081999 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:31.082007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:31.082369 (+ 362us) client_negotiation.cc:769] Sending connection context
0809 05:24:31.082622 (+ 253us) client_negotiation.cc:240] Negotiation successful
0809 05:24:31.083197 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":347,"threads_started":1}
I20260809 05:24:31.086193 11568 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:31.086512 11568 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:31.087268 11568 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:31.089772 11292 ts_manager.cc:194] Registered new tserver with Master: a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349)
I20260809 05:24:31.091871 11292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:49415
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:24:31.128554 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:31.463775 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:31.464067 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:31.464165 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:31.471903 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:31.472023 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:31.472136 11576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:31.482004 11576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:31.483286 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:31.484759 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:31.498224 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:31.498572 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:31.498849 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:31.499392 11576 server_base.cc:1034] running on GCE node
I20260809 05:24:31.500058 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:31.500998 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:31.502208 11576 hybrid_clock.cc:648] HybridClock initialized: now 1786253071502143 us; error 56 us; skew 500 ppm
I20260809 05:24:31.502534 11576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:31 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:31.505667 11576 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:31.507069 11576 webserver.cc:460] Webserver started at http://127.22.1.66:46325/ using document root <none> and password file <none>
I20260809 05:24:31.507736 11576 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:31.507853 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:31.508148 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:31.511070 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "4535f096c67c443a95389871e3551b79"
format_stamp: "Formatted at 2026-08-09 05:24:31 on dist-test-slave-rmhg"
server_key: "42ad633bf7e8675008e667f4a7023197"
server_key_iv: "4a9841e3c49e11f921aaa1ffe1337a4c"
server_key_version: "encryptionkey@0"
I20260809 05:24:31.511821 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "4535f096c67c443a95389871e3551b79"
format_stamp: "Formatted at 2026-08-09 05:24:31 on dist-test-slave-rmhg"
server_key: "42ad633bf7e8675008e667f4a7023197"
server_key_iv: "4a9841e3c49e11f921aaa1ffe1337a4c"
server_key_version: "encryptionkey@0"
I20260809 05:24:31.516943 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260809 05:24:31.520443 11599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:31.521996 11576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:31.522181 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "4535f096c67c443a95389871e3551b79"
format_stamp: "Formatted at 2026-08-09 05:24:31 on dist-test-slave-rmhg"
server_key: "42ad633bf7e8675008e667f4a7023197"
server_key_iv: "4a9841e3c49e11f921aaa1ffe1337a4c"
server_key_version: "encryptionkey@0"
I20260809 05:24:31.522367 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:31.545970 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:31.549968 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:31.550244 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:31.551298 11576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:31.552696 11576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:31.552768 11576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:31.552855 11576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:31.552897 11576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:31.584406 11576 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:36097
I20260809 05:24:31.584482 11758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:36097 every 8 connection(s)
I20260809 05:24:31.585940 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:31.595767 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 11576
I20260809 05:24:31.595971 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:24:31.596587 22533 external_mini_cluster.cc:1428] Setting key 68874911ddc24d7a22cc4dde8d281bbd
Aug 09 05:24:31 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253071, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:31.608042 11763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:31.588752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:44757 (local address 127.22.1.126:35749)
0809 05:24:31.589232 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:31.589237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:31.590453 (+ 1216us) server_negotiation.cc:407] Connection header received
0809 05:24:31.592351 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:31.592357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:31.592443 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:31.592540 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:31.594196 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.594912 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.597229 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.597915 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.598048 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:31.601775 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:31.601781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:31.601785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:31.601825 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:31.604847 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:31.605704 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:31.605710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:31.605713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:31.605871 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:31.606274 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:31.606278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:31.606280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:31.606626 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:31.606769 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:31.607483 (+ 714us) server_negotiation.cc:299] Negotiation successful
0809 05:24:31.607711 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":217,"threads_started":1}
I20260809 05:24:31.608991 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:31.589518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:44757)
0809 05:24:31.590286 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:31.590326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:31.592049 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:31.592808 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:31.592816 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:31.593417 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:31.594019 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:31.594037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.595065 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.595069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:31.597029 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:31.597045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:31.598145 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:31.598151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:31.598362 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:31.599054 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:31.599077 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:31.601527 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:31.605007 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:31.605015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:31.605031 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:31.605567 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:31.605969 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:31.605972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:31.605975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:31.606169 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:31.606755 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:31.606761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:31.607106 (+ 345us) client_negotiation.cc:769] Sending connection context
0809 05:24:31.607333 (+ 227us) client_negotiation.cc:240] Negotiation successful
0809 05:24:31.607787 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":292,"threads_started":1}
I20260809 05:24:31.610445 11760 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:31.610754 11760 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:31.611482 11760 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:31.612950 11292 ts_manager.cc:194] Registered new tserver with Master: 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
I20260809 05:24:31.613824 11292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:44757
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:24:31.659932 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:32.004364 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:32.004731 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:32.004865 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:32.012401 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:32.012547 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:32.012709 11769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:32.022715 11769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:32.024051 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:32.025457 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:32.038582 11776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:32.038543 11778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:32.038616 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:32.039516 11769 server_base.cc:1034] running on GCE node
I20260809 05:24:32.040086 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:32.041038 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:32.042249 11769 hybrid_clock.cc:648] HybridClock initialized: now 1786253072042186 us; error 60 us; skew 500 ppm
I20260809 05:24:32.042585 11769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:32 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:32.045861 11769 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:32.047396 11769 webserver.cc:460] Webserver started at http://127.22.1.67:36703/ using document root <none> and password file <none>
I20260809 05:24:32.048091 11769 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:32.048214 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:32.048520 11769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:32.051494 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "5948aff759944d1a9091325101d496b4"
format_stamp: "Formatted at 2026-08-09 05:24:32 on dist-test-slave-rmhg"
server_key: "433588f0abe330696cc15c5e3abef2ab"
server_key_iv: "d0f8a296f9d4a29c2444200b3c9d6442"
server_key_version: "encryptionkey@0"
I20260809 05:24:32.052273 11769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "5948aff759944d1a9091325101d496b4"
format_stamp: "Formatted at 2026-08-09 05:24:32 on dist-test-slave-rmhg"
server_key: "433588f0abe330696cc15c5e3abef2ab"
server_key_iv: "d0f8a296f9d4a29c2444200b3c9d6442"
server_key_version: "encryptionkey@0"
I20260809 05:24:32.057472 11769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:24:32.061172 11793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:32.062748 11769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:24:32.062897 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "5948aff759944d1a9091325101d496b4"
format_stamp: "Formatted at 2026-08-09 05:24:32 on dist-test-slave-rmhg"
server_key: "433588f0abe330696cc15c5e3abef2ab"
server_key_iv: "d0f8a296f9d4a29c2444200b3c9d6442"
server_key_version: "encryptionkey@0"
I20260809 05:24:32.063103 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:32.082757 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:32.087162 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:32.087486 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:32.088478 11769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:32.089912 11769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:32.089988 11769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:32.090109 11769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:32.090153 11769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:32.095073 11568 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:32.121304 11769 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:41637
I20260809 05:24:32.121356 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:41637 every 8 connection(s)
I20260809 05:24:32.122586 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:32.127480 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 11769
I20260809 05:24:32.127698 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:24:32.128281 22533 external_mini_cluster.cc:1428] Setting key 691fa2da81c91a4346eb76741094d881
Aug 09 05:24:32 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253072, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:32.149991 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.125526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:35751 (local address 127.22.1.126:35749)
0809 05:24:32.126044 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.126049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.127346 (+ 1297us) server_negotiation.cc:407] Connection header received
0809 05:24:32.129983 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.129989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.130092 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.130193 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:32.132799 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.134326 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.136855 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.137419 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.137545 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.141814 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:32.141821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:32.141826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:32.141872 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:32.145019 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.145977 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.145983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.145986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.146157 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.146702 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.146708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.146712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.147082 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:32.147242 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.149348 (+ 2106us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.149596 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":262,"threads_started":1}
I20260809 05:24:32.150290 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.126195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:35751)
0809 05:24:32.127061 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.127119 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.129547 (+ 2428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.130717 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.130730 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.131518 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:32.132546 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.132578 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.134586 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.134592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.136700 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.136714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.137671 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.137680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.138003 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.138729 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:32.138768 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:32.141550 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:32.145191 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:32.145202 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:32.145222 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:32.145828 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:32.146309 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:32.146314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:32.146318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:32.146564 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:32.147234 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:32.147244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:32.147749 (+ 505us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.148102 (+ 353us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.148715 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":343,"threads_started":1}
I20260809 05:24:32.151773 11959 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:32.152098 11959 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:32.153003 11959 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:32.154699 11292 ts_manager.cc:194] Registered new tserver with Master: 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637)
I20260809 05:24:32.155791 11292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:35751
I20260809 05:24:32.160900 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:24:32.180661 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.166962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38320 (local address 127.22.1.126:35749)
0809 05:24:32.167178 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.167185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.167780 (+ 595us) server_negotiation.cc:407] Connection header received
0809 05:24:32.168050 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.168055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.168127 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.168220 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:32.169802 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.170566 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.171919 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.172362 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.172476 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.175484 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:32.175492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:32.175497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:32.175543 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:32.178278 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.178902 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.178905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.178908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.179026 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.179376 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.179378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.179380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.179694 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:32.179872 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.180251 (+ 379us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.180427 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:24:32.184891 11292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:24:32.191797 11292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:24:32.214561 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.203870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59666 (local address 127.22.1.67:41637)
0809 05:24:32.204670 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.204678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.204716 (+ 38us) server_negotiation.cc:407] Connection header received
0809 05:24:32.205059 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.205068 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.205356 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.205552 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:32.206629 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.208394 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.211798 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213183 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.213301 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.213559 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.214044 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.214219 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":864,"threads_started":1}
I20260809 05:24:32.214697 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.204462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36097 (local address 127.0.0.1:57282)
0809 05:24:32.205331 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.205349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.205480 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.206562 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.206568 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.206596 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:32.207040 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.207052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.209311 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.209321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.211709 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.211729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213632 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.213639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.213903 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.213938 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.214020 (+ 82us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.214200 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":116,"thread_start_us":780,"threads_started":1}
I20260809 05:24:32.214730 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.203748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:41637 (local address 127.0.0.1:59666)
0809 05:24:32.204569 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.204610 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.204774 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.205885 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.205893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.205926 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:32.206396 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.206408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.208565 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.208572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.211593 (+ 3021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.211608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213380 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.213386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.213570 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.213684 (+ 114us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.213829 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.214140 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":128,"thread_start_us":413,"threads_started":1}
I20260809 05:24:32.215711 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.204999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.22.1.66:36097)
0809 05:24:32.205959 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.205968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.205994 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:24:32.206093 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.206102 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.206328 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.207245 (+ 917us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:32.207503 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.209171 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.211887 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213371 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.213709 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.213914 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.215216 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.215397 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":846,"threads_started":1}
I20260809 05:24:32.219239 11873 tablet_service.cc:1467] Processing CreateTablet for tablet 4f387d021bfe4672b5b40a773781cd35 (DEFAULT_TABLE table=test-table [id=f3a099e74f2d44518641433f9ca887e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:32.219239 11673 tablet_service.cc:1467] Processing CreateTablet for tablet 4f387d021bfe4672b5b40a773781cd35 (DEFAULT_TABLE table=test-table [id=f3a099e74f2d44518641433f9ca887e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:32.220788 11673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f387d021bfe4672b5b40a773781cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:32.220815 11873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f387d021bfe4672b5b40a773781cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:32.222671 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.204996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50908)
0809 05:24:32.210381 (+ 5385us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.210400 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.210556 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.211146 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.211152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.211176 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:32.211690 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.211701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213544 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.213550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.215581 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.215596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.221794 (+ 6198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.221802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.222006 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.222043 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.222134 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.222275 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5257,"mutex_wait_us":8,"thread_start_us":5116,"threads_started":1}
I20260809 05:24:32.222703 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.205695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50908 (local address 127.22.1.65:33349)
0809 05:24:32.209943 (+ 4248us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.209950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.210523 (+ 573us) server_negotiation.cc:407] Connection header received
0809 05:24:32.210720 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.210727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.210954 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.211136 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:32.211846 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.213386 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.215769 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.221582 (+ 5813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.221720 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.221976 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.222196 (+ 220us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.222369 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4093,"thread_start_us":3923,"threads_started":1}
I20260809 05:24:32.226630 11479 tablet_service.cc:1467] Processing CreateTablet for tablet 4f387d021bfe4672b5b40a773781cd35 (DEFAULT_TABLE table=test-table [id=f3a099e74f2d44518641433f9ca887e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:32.227963 11479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f387d021bfe4672b5b40a773781cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:32.235543 11986 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Bootstrap starting.
I20260809 05:24:32.237718 11989 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Bootstrap starting.
I20260809 05:24:32.239293 11986 tablet_bootstrap.cc:654] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:32.240628 11986 log.cc:824] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:32.241371 11989 tablet_bootstrap.cc:654] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:32.241720 11990 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Bootstrap starting.
I20260809 05:24:32.242687 11989 log.cc:824] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:32.244794 11990 tablet_bootstrap.cc:654] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:32.245754 11990 log.cc:824] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:32.246140 11989 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: No bootstrap required, opened a new log
I20260809 05:24:32.246428 11989 ts_tablet_manager.cc:1397] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260809 05:24:32.248984 11990 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: No bootstrap required, opened a new log
I20260809 05:24:32.249037 11986 tablet_bootstrap.cc:492] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: No bootstrap required, opened a new log
I20260809 05:24:32.249291 11990 ts_tablet_manager.cc:1397] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260809 05:24:32.249323 11986 ts_tablet_manager.cc:1397] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20260809 05:24:32.256069 11986 raft_consensus.cc:348] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.256428 11986 raft_consensus.cc:374] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:32.256528 11986 raft_consensus.cc:729] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948aff759944d1a9091325101d496b4, State: Initialized, Role: FOLLOWER
I20260809 05:24:32.256552 11989 raft_consensus.cc:348] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.256956 11989 raft_consensus.cc:374] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:32.257050 11989 raft_consensus.cc:729] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4535f096c67c443a95389871e3551b79, State: Initialized, Role: FOLLOWER
I20260809 05:24:32.257134 11986 consensus_queue.cc:260] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.257647 11989 consensus_queue.cc:260] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.258371 11959 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:32.259099 11986 ts_tablet_manager.cc:1428] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260809 05:24:32.259366 11990 raft_consensus.cc:348] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.259769 11990 raft_consensus.cc:374] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:32.259851 11990 raft_consensus.cc:729] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9260b59fdb9487b8d39b5605082af51, State: Initialized, Role: FOLLOWER
I20260809 05:24:32.260456 11990 consensus_queue.cc:260] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.262850 11990 ts_tablet_manager.cc:1428] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260809 05:24:32.264322 11760 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:32.265192 11989 ts_tablet_manager.cc:1428] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
W20260809 05:24:32.314952 11570 tablet.cc:2369] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:24:32.337990 11761 tablet.cc:2369] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:24:32.374181 11960 tablet.cc:2369] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:32.408951 12000 raft_consensus.cc:482] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:24:32.409303 12000 raft_consensus.cc:504] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.411088 12000 leader_election.cc:288] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349), 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
I20260809 05:24:32.416899 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.411435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36097 (local address 127.22.1.67:47707)
0809 05:24:32.411677 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.411698 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.411830 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.412296 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.412301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.412383 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:32.412594 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.412605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.413656 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.413660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.414930 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.414941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.416150 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.416155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.416342 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.416387 (+ 45us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.416476 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.416617 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"spinlock_wait_cycles":25344}
I20260809 05:24:32.416906 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.411840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47707 (local address 127.22.1.66:36097)
0809 05:24:32.412033 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.412039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.412054 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:24:32.412115 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.412119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.412174 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.412250 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:32.412731 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.413519 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.415087 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.415990 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.416128 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.416225 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.416522 (+ 297us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.416653 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:24:32.418073 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.411437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47317 (local address 127.22.1.65:33349)
0809 05:24:32.411687 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.411693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.412317 (+ 624us) server_negotiation.cc:407] Connection header received
0809 05:24:32.412482 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.412487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.412550 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.412635 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:32.413191 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.413903 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.415783 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.416850 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.416982 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.417088 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.417693 (+ 605us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.417835 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260809 05:24:32.418084 12008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.411468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.22.1.67:47317)
0809 05:24:32.412198 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:32.412213 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:32.412345 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:32.412656 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:32.412660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:32.412684 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:32.413006 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.413016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.414063 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.414066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.415619 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:32.415634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.417336 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.417342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:32.417478 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.417502 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 05:24:32.417641 (+ 139us) client_negotiation.cc:240] Negotiation successful
0809 05:24:32.417766 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":843,"threads_started":1}
I20260809 05:24:32.418143 11704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f387d021bfe4672b5b40a773781cd35" candidate_uuid: "5948aff759944d1a9091325101d496b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4535f096c67c443a95389871e3551b79" is_pre_election: true
I20260809 05:24:32.418751 11704 raft_consensus.cc:2454] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5948aff759944d1a9091325101d496b4 in term 0.
I20260809 05:24:32.419209 11505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f387d021bfe4672b5b40a773781cd35" candidate_uuid: "5948aff759944d1a9091325101d496b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9260b59fdb9487b8d39b5605082af51" is_pre_election: true
I20260809 05:24:32.419512 11797 leader_election.cc:302] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4535f096c67c443a95389871e3551b79, 5948aff759944d1a9091325101d496b4; no voters:
I20260809 05:24:32.419787 11505 raft_consensus.cc:2454] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5948aff759944d1a9091325101d496b4 in term 0.
I20260809 05:24:32.419957 12000 raft_consensus.cc:2781] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:24:32.420132 12000 raft_consensus.cc:482] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:24:32.420392 12000 raft_consensus.cc:3037] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:32.423583 12000 raft_consensus.cc:504] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.424345 12000 leader_election.cc:288] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [CANDIDATE]: Term 1 election: Requested vote from peers a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349), 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
I20260809 05:24:32.424861 11505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f387d021bfe4672b5b40a773781cd35" candidate_uuid: "5948aff759944d1a9091325101d496b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9260b59fdb9487b8d39b5605082af51"
I20260809 05:24:32.424906 11704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f387d021bfe4672b5b40a773781cd35" candidate_uuid: "5948aff759944d1a9091325101d496b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4535f096c67c443a95389871e3551b79"
I20260809 05:24:32.425072 11505 raft_consensus.cc:3037] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:32.425107 11704 raft_consensus.cc:3037] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:32.427518 11704 raft_consensus.cc:2454] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5948aff759944d1a9091325101d496b4 in term 1.
I20260809 05:24:32.427945 11797 leader_election.cc:302] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4535f096c67c443a95389871e3551b79, 5948aff759944d1a9091325101d496b4; no voters:
I20260809 05:24:32.428226 12000 raft_consensus.cc:2781] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:32.428328 11505 raft_consensus.cc:2454] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5948aff759944d1a9091325101d496b4 in term 1.
I20260809 05:24:32.428735 12000 raft_consensus.cc:686] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5948aff759944d1a9091325101d496b4, State: Running, Role: LEADER
I20260809 05:24:32.429318 12000 consensus_queue.cc:237] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:32.433944 11292 catalog_manager.cc:5495] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5948aff759944d1a9091325101d496b4 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "5948aff759944d1a9091325101d496b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } health_report { overall_health: UNKNOWN } } }
I20260809 05:24:32.461299 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.456084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.22.1.67:41637)
0809 05:24:32.456255 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.456260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.456327 (+ 67us) server_negotiation.cc:407] Connection header received
0809 05:24:32.456546 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.456550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.456651 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.456775 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:24:32.457097 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.457876 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.459317 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.459718 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.459813 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.460251 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:24:32.460787 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:24:32.460895 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.460977 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.461045 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:24:32.469890 11704 raft_consensus.cc:1261] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Refusing update from remote peer 5948aff759944d1a9091325101d496b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:32.469887 11505 raft_consensus.cc:1261] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Refusing update from remote peer 5948aff759944d1a9091325101d496b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:32.470860 12000 consensus_queue.cc:1035] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:32.471145 12010 consensus_queue.cc:1035] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:32.481616 12017 mvcc.cc:205] Tried to move back new op lower bound from 7316492584826052608 to 7316492584674308096. Current Snapshot: MvccSnapshot[applied={T|T < 7316492584826052608}]
I20260809 05:24:32.496328 11291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:32.496701 11291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:32.500624 11291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Sending DeleteTablet for 3 replicas of tablet 4f387d021bfe4672b5b40a773781cd35
I20260809 05:24:32.501830 11873 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f387d021bfe4672b5b40a773781cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:32 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:59666
I20260809 05:24:32.501827 11479 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f387d021bfe4672b5b40a773781cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:32 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:50908
I20260809 05:24:32.501919 11673 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f387d021bfe4672b5b40a773781cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:32 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:57282
I20260809 05:24:32.503077 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 11388
I20260809 05:24:32.514251 12030 tablet_replica.cc:330] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: stopping tablet replica
I20260809 05:24:32.514317 12031 tablet_replica.cc:330] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: stopping tablet replica
I20260809 05:24:32.514798 12031 raft_consensus.cc:2229] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:32.514808 12030 raft_consensus.cc:2229] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:32.515345 12031 raft_consensus.cc:2258] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:32.515350 12030 raft_consensus.cc:2258] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:32.516407 12029 tablet_replica.cc:330] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: stopping tablet replica
I20260809 05:24:32.516870 12029 raft_consensus.cc:2229] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:24:32.517740 12031 ts_tablet_manager.cc:1905] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:32.517745 12029 raft_consensus.cc:2258] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:32.520067 12029 ts_tablet_manager.cc:1905] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:32.524879 12031 ts_tablet_manager.cc:1918] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:32.525070 12031 log.cc:1196] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/4f387d021bfe4672b5b40a773781cd35
I20260809 05:24:32.525779 12031 ts_tablet_manager.cc:1939] T 4f387d021bfe4672b5b40a773781cd35 P 4535f096c67c443a95389871e3551b79: Deleting consensus metadata
I20260809 05:24:32.527415 12029 ts_tablet_manager.cc:1918] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:32.527606 12029 log.cc:1196] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/4f387d021bfe4672b5b40a773781cd35
I20260809 05:24:32.528005 11260 catalog_manager.cc:4841] TS 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097): tablet 4f387d021bfe4672b5b40a773781cd35 (table test-table [id=f3a099e74f2d44518641433f9ca887e7]) successfully deleted
I20260809 05:24:32.528196 12029 ts_tablet_manager.cc:1939] T 4f387d021bfe4672b5b40a773781cd35 P 5948aff759944d1a9091325101d496b4: Deleting consensus metadata
I20260809 05:24:32.530282 11261 catalog_manager.cc:4841] TS 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637): tablet 4f387d021bfe4672b5b40a773781cd35 (table test-table [id=f3a099e74f2d44518641433f9ca887e7]) successfully deleted
I20260809 05:24:32.530509 12030 ts_tablet_manager.cc:1905] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:32.536491 12030 ts_tablet_manager.cc:1918] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Aug 09 05:24:32 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:24:32.536666 12030 log.cc:1196] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/4f387d021bfe4672b5b40a773781cd35
I20260809 05:24:32.537110 12030 ts_tablet_manager.cc:1939] T 4f387d021bfe4672b5b40a773781cd35 P a9260b59fdb9487b8d39b5605082af51: Deleting consensus metadata
I20260809 05:24:32.539001 11261 catalog_manager.cc:4841] TS a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349): tablet 4f387d021bfe4672b5b40a773781cd35 (table test-table [id=f3a099e74f2d44518641433f9ca887e7]) successfully deleted
I20260809 05:24:32.544615 11985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.504333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50920 (local address 127.22.1.65:33349)
0809 05:24:32.504545 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.504552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.520890 (+ 16338us) server_negotiation.cc:407] Connection header received
0809 05:24:32.521244 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.521250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.521335 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.521409 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:32.523488 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.524563 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.527211 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.527807 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.527935 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.538074 (+ 10139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:32.538090 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:32.538109 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:32.538158 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:32.541377 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.542448 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.542456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.542459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.542631 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.543052 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.543055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.543057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.543447 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:32.543726 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.544084 (+ 358us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.544272 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:24:32.545346 11559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:32.654470 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 11388
I20260809 05:24:32.679675 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 11576
Aug 09 05:24:32 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:24:32.695741 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.680619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.22.1.66:36097)
0809 05:24:32.680786 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.680790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.680809 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:24:32.681016 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.681021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.681130 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.681254 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:32.682913 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.683728 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.685254 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.685654 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.685766 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.690041 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:32.690057 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:32.690075 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:32.690110 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:32.692885 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.693601 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.693610 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.693612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.693743 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.694137 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.694141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.694143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.694626 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:32.694918 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.695278 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.695452 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:24:32.696470 11753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:32.801913 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 11576
I20260809 05:24:32.828604 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 11769
Aug 09 05:24:32 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:24:32.847525 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:32.829455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.22.1.67:41637)
0809 05:24:32.829640 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:32.829646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:32.829720 (+ 74us) server_negotiation.cc:407] Connection header received
0809 05:24:32.829955 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:32.829964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:32.830052 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:32.830142 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:32.831529 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.832456 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.833853 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:32.834333 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:32.834466 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:32.840296 (+ 5830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:32.840313 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:32.840333 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:32.840398 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:32.843629 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.844614 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.844626 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.844630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.844862 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:32.845360 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:32.845366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:32.845369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:32.845922 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:32.846296 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:32.846781 (+ 485us) server_negotiation.cc:299] Negotiation successful
0809 05:24:32.847065 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 05:24:32.848337 11954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:32.972994 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 11769
I20260809 05:24:33.002292 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 11240
I20260809 05:24:33.003449 11340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:33.111153 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 11240
I20260809 05:24:33.134317 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--webserver_interface=127.22.1.126
--webserver_port=44691
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:33.473677 12038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:33.474020 12038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:33.474157 12038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:33.481671 12038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:33.481812 12038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:33.481896 12038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:33.481974 12038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:33.482038 12038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:33.492404 12038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:33.493942 12038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:33.495352 12038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:33.519359 12046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:33.519384 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:33.519318 12052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:33.520591 12038 server_base.cc:1034] running on GCE node
I20260809 05:24:33.521606 12038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:33.523270 12038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:33.524497 12038 hybrid_clock.cc:648] HybridClock initialized: now 1786253073524434 us; error 52 us; skew 500 ppm
I20260809 05:24:33.524925 12038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:33 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253073, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:33.532009 12038 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:33.533876 12038 webserver.cc:460] Webserver started at http://127.22.1.126:44691/ using document root <none> and password file <none>
I20260809 05:24:33.534694 12038 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:33.534843 12038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:33.541409 12038 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260809 05:24:33.544639 12061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:33.546267 12038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:33.546471 12038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "3c8bdf3527a243828957d46269780406"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "1262517ee6b3cd91889563cd9621f700"
server_key_iv: "cd46f5013f8c40430de9994b3130ec34"
server_key_version: "encryptionkey@0"
I20260809 05:24:33.547600 12038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:33.581387 12038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:33.586282 12038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:33.586611 12038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:33.605315 12038 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35749
I20260809 05:24:33.605343 12132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35749 every 8 connection(s)
I20260809 05:24:33.606931 12038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:33.613499 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12038
I20260809 05:24:33.614552 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:33349
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=37299
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:33.614979 12133 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:24:33.619807 12133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap starting.
I20260809 05:24:33.641016 12133 log.cc:824] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:33.654273 12133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:24:33.654850 12133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap complete.
I20260809 05:24:33.664106 12133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:33.664978 12133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8bdf3527a243828957d46269780406, State: Initialized, Role: FOLLOWER
I20260809 05:24:33.665683 12133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:33.665915 12133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:33.666051 12133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:33.666206 12133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:24:33.670161 12133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:33.670718 12133 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8bdf3527a243828957d46269780406; no voters:
I20260809 05:24:33.671340 12133 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 05:24:33.671502 12139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Leader election won for term 2
I20260809 05:24:33.672773 12139 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 LEADER]: Becoming Leader. State: Replica: 3c8bdf3527a243828957d46269780406, State: Running, Role: LEADER
I20260809 05:24:33.673384 12139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:33.673812 12133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:33.676035 12140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:33.676498 12140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:33.676877 12142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8bdf3527a243828957d46269780406. Latest consensus state: current_term: 2 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:33.677614 12142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:33.682327 12145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:33.692106 12145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:33.693709 12145 catalog_manager.cc:1259] Loaded cluster ID: a9905725d5134e36b39b230db67b8f1e
I20260809 05:24:33.693897 12145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:33.697383 12145 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260809 05:24:33.714219 12145 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260809 05:24:33.723506 12145 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:33.725322 12145 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Loaded TSK: 0
I20260809 05:24:33.727011 12145 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 05:24:33.979974 12136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:33.980260 12136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:33.980381 12136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:33.987906 12136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:33.988030 12136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:33.988180 12136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:33.998342 12136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:33349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=37299
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:33.999719 12136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:34.001118 12136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:34.014753 12172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.014842 12174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.015374 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:34.016602 12136 server_base.cc:1034] running on GCE node
I20260809 05:24:34.017343 12136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:34.018381 12136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:34.019605 12136 hybrid_clock.cc:648] HybridClock initialized: now 1786253074019554 us; error 44 us; skew 500 ppm
I20260809 05:24:34.019922 12136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:34 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:34.023190 12136 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:34.024762 12136 webserver.cc:460] Webserver started at http://127.22.1.65:37299/ using document root <none> and password file <none>
I20260809 05:24:34.025487 12136 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:34.025633 12136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:34.031020 12136 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:24:34.033814 12183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.035238 12136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:34.035392 12136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "a9260b59fdb9487b8d39b5605082af51"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "4835b7ceb2ec78124869af9eb30b4b08"
server_key_iv: "e9e54f3d1432cd56839b6cc7914531ef"
server_key_version: "encryptionkey@0"
I20260809 05:24:34.036374 12136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:34.071271 12136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:34.075693 12136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:34.075958 12136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:34.077001 12136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:34.078410 12136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:34.078516 12136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.078627 12136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:34.078706 12136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.110746 12136 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:33349
I20260809 05:24:34.110823 12335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:33349 every 8 connection(s)
I20260809 05:24:34.112308 12136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:34.115513 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12136
I20260809 05:24:34.116338 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:36097
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=46325
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:34 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:34.136343 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:34.115361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:40877 (local address 127.22.1.126:35749)
0809 05:24:34.116072 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:34.116085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:34.116847 (+ 762us) server_negotiation.cc:407] Connection header received
0809 05:24:34.118975 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:34.118998 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:34.119340 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:34.119700 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:34.121328 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.122555 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.124158 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.124865 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.125077 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:34.128544 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:34.128553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:34.128569 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:34.128607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:34.131481 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:34.132249 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:34.132257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:34.132262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:34.132446 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:34.132886 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:34.132891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:34.132894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:34.133441 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:34.133650 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:34.134459 (+ 809us) server_negotiation.cc:299] Negotiation successful
0809 05:24:34.134932 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":385,"threads_started":1}
I20260809 05:24:34.136346 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:34.115762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:40877)
0809 05:24:34.116646 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:34.116704 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:34.118433 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:34.119863 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:34.119874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:34.120521 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:34.121079 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:34.121098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.122725 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.122729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:34.124012 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:34.124026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.125121 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.125129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:34.125422 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:34.126061 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:34.126091 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:34.128264 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:34.131633 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:34.131642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:34.131659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:34.132100 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:34.132565 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:34.132571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:34.132575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:34.132768 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:34.133590 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:34.133599 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:34.133958 (+ 359us) client_negotiation.cc:769] Sending connection context
0809 05:24:34.134210 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:24:34.134977 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":342,"threads_started":1}
I20260809 05:24:34.139829 12336 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:34.140141 12336 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:34.140928 12336 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:34.143406 12085 ts_manager.cc:194] Registered new tserver with Master: a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349)
I20260809 05:24:34.145509 12085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:40877
W20260809 05:24:34.462697 12341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:34.463030 12341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:34.463138 12341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:34.470427 12341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:34.470541 12341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:34.470667 12341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:34.480460 12341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:36097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=46325
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:34.481864 12341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:34.483155 12341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:34.501066 12349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.501140 12354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.501188 12356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:34.501967 12341 server_base.cc:1034] running on GCE node
I20260809 05:24:34.502660 12341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:34.503736 12341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:34.504942 12341 hybrid_clock.cc:648] HybridClock initialized: now 1786253074504888 us; error 49 us; skew 500 ppm
I20260809 05:24:34.505417 12341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:34 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:34.508742 12341 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:34.510422 12341 webserver.cc:460] Webserver started at http://127.22.1.66:46325/ using document root <none> and password file <none>
I20260809 05:24:34.511170 12341 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:34.511318 12341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:34.517047 12341 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 05:24:34.519979 12366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.521410 12341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:34.521592 12341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "4535f096c67c443a95389871e3551b79"
format_stamp: "Formatted at 2026-08-09 05:24:31 on dist-test-slave-rmhg"
server_key: "42ad633bf7e8675008e667f4a7023197"
server_key_iv: "4a9841e3c49e11f921aaa1ffe1337a4c"
server_key_version: "encryptionkey@0"
I20260809 05:24:34.522639 12341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:34.537424 12341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:34.541775 12341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:34.542050 12341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:34.543128 12341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:34.544510 12341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:34.544615 12341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.544718 12341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:34.544787 12341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.576748 12341 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:36097
I20260809 05:24:34.576807 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:36097 every 8 connection(s)
I20260809 05:24:34.578275 12341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:34.583389 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12341
I20260809 05:24:34.584010 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:41637
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=36703
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:24:34 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:34.599229 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:34.580805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:38155 (local address 127.22.1.126:35749)
0809 05:24:34.581014 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:34.581020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:34.582288 (+ 1268us) server_negotiation.cc:407] Connection header received
0809 05:24:34.584187 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:34.584193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:34.584301 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:34.584407 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:34.586063 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.587036 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.588675 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.589324 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.589449 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:34.593259 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:34.593267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:34.593271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:34.593318 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:34.595894 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:34.596834 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:34.596838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:34.596840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:34.596981 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:34.597484 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:34.597487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:34.597489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:34.597765 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:34.597875 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:34.598773 (+ 898us) server_negotiation.cc:299] Negotiation successful
0809 05:24:34.598961 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 05:24:34.600704 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:34.581376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:38155)
0809 05:24:34.582111 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:34.582158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:34.583874 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:34.584660 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:34.584669 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:34.585261 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:34.585872 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:34.585890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.587201 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.587205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:34.588514 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:34.588532 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:34.589553 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:34.589560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:34.589779 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:34.590481 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:34.590504 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:34.592923 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:34.596071 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:34.596081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:34.596101 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:34.596698 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:34.597102 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:34.597106 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:34.597110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:34.597384 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:34.597890 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:34.597898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:34.598386 (+ 488us) client_negotiation.cc:769] Sending connection context
0809 05:24:34.598688 (+ 302us) client_negotiation.cc:240] Negotiation successful
0809 05:24:34.599258 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":275,"threads_started":1}
I20260809 05:24:34.602257 12540 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:34.602607 12540 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:34.603381 12540 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:34.605064 12085 ts_manager.cc:194] Registered new tserver with Master: 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
I20260809 05:24:34.606294 12085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:38155
W20260809 05:24:34.927495 12543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:34.927830 12543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:34.927934 12543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:34.935175 12543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:34.935295 12543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:34.935417 12543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:34.945518 12543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:41637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=36703
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:34.946820 12543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:34.948143 12543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:34.961404 12549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.961427 12554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:34.961364 12548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:34.962045 12543 server_base.cc:1034] running on GCE node
I20260809 05:24:34.962656 12543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:34.963663 12543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:34.964871 12543 hybrid_clock.cc:648] HybridClock initialized: now 1786253074964821 us; error 41 us; skew 500 ppm
I20260809 05:24:34.965226 12543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:34 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:34.968340 12543 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:34.969812 12543 webserver.cc:460] Webserver started at http://127.22.1.67:36703/ using document root <none> and password file <none>
I20260809 05:24:34.970520 12543 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:34.970654 12543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:34.976128 12543 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 05:24:34.978875 12565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:34.980334 12543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:24:34.980492 12543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "5948aff759944d1a9091325101d496b4"
format_stamp: "Formatted at 2026-08-09 05:24:32 on dist-test-slave-rmhg"
server_key: "433588f0abe330696cc15c5e3abef2ab"
server_key_iv: "d0f8a296f9d4a29c2444200b3c9d6442"
server_key_version: "encryptionkey@0"
I20260809 05:24:34.981510 12543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:34.998260 12543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:35.002283 12543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:35.002548 12543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:35.003548 12543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:35.004976 12543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:35.005086 12543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:35.005223 12543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:35.005300 12543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:35.036974 12543 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:41637
I20260809 05:24:35.037029 12736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:41637 every 8 connection(s)
I20260809 05:24:35.038471 12543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:35.044073 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12543
Aug 09 05:24:35 dist-test-slave-rmhg krb5kdc[11224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253074, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:35.060707 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.041308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:46191 (local address 127.22.1.126:35749)
0809 05:24:35.041523 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.041528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.042747 (+ 1219us) server_negotiation.cc:407] Connection header received
0809 05:24:35.044641 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.044647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.044731 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.044821 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:35.046689 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.047423 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.049485 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.049938 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.050034 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.054413 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:35.054421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:35.054425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:35.054470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:35.057400 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.058177 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.058182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.058186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.058354 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.058785 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.058790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.058794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.059179 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:35.059330 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.060237 (+ 907us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.060452 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:24:35.062145 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.045108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.22.1.126:35749)
0809 05:24:35.048002 (+ 2894us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.048006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.048023 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:24:35.048064 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.048067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.048118 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.048204 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:35.050206 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.050931 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.052876 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.053533 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.053636 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.056249 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:35.056257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:35.056262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:35.056312 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:35.059416 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.060226 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.060230 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.060233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.060358 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.060801 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.060806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.060809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.061100 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:35.061323 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.061695 (+ 372us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.061850 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2798,"thread_start_us":2654,"threads_started":1}
I20260809 05:24:35.062206 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.041826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:46191)
0809 05:24:35.042585 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.042632 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.044339 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.045098 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.045107 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.045909 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:35.046517 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.046535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.047588 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.047592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.049343 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.049360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.050179 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.050187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.050411 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.051076 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:35.051145 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:35.054162 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:35.057557 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:35.057565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:35.057580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:35.058039 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:35.058460 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:35.058464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:35.058466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:35.058657 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:35.059330 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:35.059336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:35.059784 (+ 448us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.060146 (+ 362us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.060676 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":292,"threads_started":1}
I20260809 05:24:35.063741 12737 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35749
I20260809 05:24:35.064100 12737 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:35.064978 12737 heartbeater.cc:511] Master 127.22.1.126:35749 requested a full tablet report, sending...
I20260809 05:24:35.066517 12085 ts_manager.cc:194] Registered new tserver with Master: 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637)
I20260809 05:24:35.067312 12085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:46191
I20260809 05:24:35.077291 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:24:35.083935 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.078208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.22.1.126:35749)
0809 05:24:35.078355 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.078359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.078795 (+ 436us) server_negotiation.cc:407] Connection header received
0809 05:24:35.078984 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.078988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.079149 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.079273 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:24:35.079660 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.080403 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.081906 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.082323 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.082413 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.082903 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:24:35.083419 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:24:35.083547 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.083651 (+ 104us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.083715 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:24:35.084908 12085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:24:35.092289 12085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:24:35.115837 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.103926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50928)
0809 05:24:35.105905 (+ 1979us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.105950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.106147 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.106863 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.106870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.106898 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:35.107233 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.107244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.109137 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.109142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.112799 (+ 3657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.112815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.114783 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.114794 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.114997 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.115072 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.115186 (+ 114us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.115373 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"mutex_wait_us":24,"thread_start_us":2561,"threads_started":1}
I20260809 05:24:35.116559 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.104811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57306 (local address 127.22.1.66:36097)
0809 05:24:35.105372 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.105382 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.106955 (+ 1573us) server_negotiation.cc:407] Connection header received
0809 05:24:35.107060 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.107067 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.107287 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.107487 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:35.108012 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.109480 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.113259 (+ 3779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.114601 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.114716 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.114923 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.116084 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.116262 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":333,"threads_started":1}
I20260809 05:24:35.116753 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.104424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36097 (local address 127.0.0.1:57306)
0809 05:24:35.106718 (+ 2294us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.106734 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.106838 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.107451 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.107455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.107480 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:35.107824 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.107836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.109650 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.109659 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.113062 (+ 3403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.113077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.115107 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.115114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.115331 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.115362 (+ 31us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.115451 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.115586 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2168,"mutex_wait_us":47,"thread_start_us":1036,"threads_started":1}
I20260809 05:24:35.117362 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.104209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.22.1.65:33349)
0809 05:24:35.104806 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.104816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.106077 (+ 1261us) server_negotiation.cc:407] Connection header received
0809 05:24:35.106386 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.106395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.106643 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.106849 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:35.107385 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.108964 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.112991 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.114527 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.116280 (+ 1753us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.116506 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.116578 (+ 72us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.116933 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":723,"threads_started":1}
I20260809 05:24:35.120261 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.107612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:41637 (local address 127.0.0.1:59688)
0809 05:24:35.108479 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.108493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.108597 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.109448 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.109452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.109478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:35.111316 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.111331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.114438 (+ 3107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.114443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.117682 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.117697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.119445 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.119452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.119623 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.119657 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.119755 (+ 98us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.119893 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1,"thread_start_us":654,"threads_started":1}
I20260809 05:24:35.120316 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.107224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.22.1.67:41637)
0809 05:24:35.108097 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.108106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.108585 (+ 479us) server_negotiation.cc:407] Connection header received
0809 05:24:35.108756 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.108764 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.108978 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.109182 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:35.111546 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.114178 (+ 2632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.117829 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.119219 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.119355 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.119575 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.119813 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.119983 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":573,"threads_started":1}
I20260809 05:24:35.120819 12254 tablet_service.cc:1467] Processing CreateTablet for tablet 83878663871141f7b2bde397d38d8729 (DEFAULT_TABLE table=test-table [id=543ee424ff59407ab927e258972451af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:35.120828 12443 tablet_service.cc:1467] Processing CreateTablet for tablet 83878663871141f7b2bde397d38d8729 (DEFAULT_TABLE table=test-table [id=543ee424ff59407ab927e258972451af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:35.122511 12443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83878663871141f7b2bde397d38d8729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:35.122532 12254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83878663871141f7b2bde397d38d8729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:35.124185 12638 tablet_service.cc:1467] Processing CreateTablet for tablet 83878663871141f7b2bde397d38d8729 (DEFAULT_TABLE table=test-table [id=543ee424ff59407ab927e258972451af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:35.125692 12638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83878663871141f7b2bde397d38d8729. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:35.135409 12760 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Bootstrap starting.
I20260809 05:24:35.138602 12761 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Bootstrap starting.
I20260809 05:24:35.139338 12760 tablet_bootstrap.cc:654] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:35.140784 12760 log.cc:824] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:35.141040 12762 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Bootstrap starting.
I20260809 05:24:35.142836 12761 tablet_bootstrap.cc:654] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:35.144120 12762 tablet_bootstrap.cc:654] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:35.144248 12761 log.cc:824] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:35.145401 12762 log.cc:824] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:35.147454 12760 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: No bootstrap required, opened a new log
I20260809 05:24:35.147789 12760 ts_tablet_manager.cc:1397] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260809 05:24:35.148294 12762 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: No bootstrap required, opened a new log
I20260809 05:24:35.148361 12761 tablet_bootstrap.cc:492] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: No bootstrap required, opened a new log
I20260809 05:24:35.148547 12762 ts_tablet_manager.cc:1397] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 05:24:35.148638 12761 ts_tablet_manager.cc:1397] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 05:24:35.148803 12336 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:35.158020 12762 raft_consensus.cc:348] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.158398 12762 raft_consensus.cc:374] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:35.158493 12762 raft_consensus.cc:729] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948aff759944d1a9091325101d496b4, State: Initialized, Role: FOLLOWER
I20260809 05:24:35.158641 12760 raft_consensus.cc:348] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.159104 12760 raft_consensus.cc:374] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:35.159205 12760 raft_consensus.cc:729] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4535f096c67c443a95389871e3551b79, State: Initialized, Role: FOLLOWER
I20260809 05:24:35.159091 12762 consensus_queue.cc:260] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.159091 12761 raft_consensus.cc:348] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.159504 12761 raft_consensus.cc:374] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:35.159619 12761 raft_consensus.cc:729] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9260b59fdb9487b8d39b5605082af51, State: Initialized, Role: FOLLOWER
I20260809 05:24:35.159821 12760 consensus_queue.cc:260] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.160262 12761 consensus_queue.cc:260] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.160481 12737 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:35.161118 12762 ts_tablet_manager.cc:1428] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260809 05:24:35.162416 12540 heartbeater.cc:503] Master 127.22.1.126:35749 was elected leader, sending a full tablet report...
I20260809 05:24:35.162830 12760 ts_tablet_manager.cc:1428] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260809 05:24:35.163189 12761 ts_tablet_manager.cc:1428] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260809 05:24:35.290177 12738 tablet.cc:2369] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:35.322880 12770 raft_consensus.cc:482] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:24:35.323203 12770 raft_consensus.cc:504] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.324944 12770 leader_election.cc:288] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637), 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
W20260809 05:24:35.330288 12541 tablet.cc:2369] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:35.332657 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.325348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:60109 (local address 127.22.1.67:41637)
0809 05:24:35.325559 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.325565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.326351 (+ 786us) server_negotiation.cc:407] Connection header received
0809 05:24:35.326546 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.326552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.326634 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.326709 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:35.327425 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.328212 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.330165 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.331409 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.331567 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.331678 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.332223 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.332367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:24:35.332643 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.325677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:57871 (local address 127.22.1.66:36097)
0809 05:24:35.325883 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.325889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.326521 (+ 632us) server_negotiation.cc:407] Connection header received
0809 05:24:35.326663 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.326666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.326737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.326826 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:35.327610 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.328316 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.330243 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.331458 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.331592 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.331678 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.332223 (+ 545us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.332367 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 05:24:35.332818 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.325432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36097 (local address 127.22.1.65:57871)
0809 05:24:35.326403 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.326417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.326548 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.326913 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.326919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.326967 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:35.327433 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.327447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.328455 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.328459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.330005 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.330028 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.331730 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.331735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.331958 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.332051 (+ 93us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.332154 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.332343 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":104,"spinlock_wait_cycles":896,"thread_start_us":637,"threads_started":1}
I20260809 05:24:35.333204 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.325264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:41637 (local address 127.22.1.65:60109)
0809 05:24:35.326222 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.326242 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:35.326400 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:35.326745 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:35.326750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:35.326774 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:35.327270 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.327281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.328349 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.328354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.330009 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:35.330021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.331715 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.331722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:35.331975 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.332058 (+ 83us) client_negotiation.cc:769] Sending connection context
0809 05:24:35.332154 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 05:24:35.332343 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":164,"spinlock_wait_cycles":33536,"thread_start_us":468,"threads_started":1}
I20260809 05:24:35.333909 12469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83878663871141f7b2bde397d38d8729" candidate_uuid: "a9260b59fdb9487b8d39b5605082af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4535f096c67c443a95389871e3551b79" is_pre_election: true
I20260809 05:24:35.334156 12666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83878663871141f7b2bde397d38d8729" candidate_uuid: "a9260b59fdb9487b8d39b5605082af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5948aff759944d1a9091325101d496b4" is_pre_election: true
I20260809 05:24:35.334542 12469 raft_consensus.cc:2454] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9260b59fdb9487b8d39b5605082af51 in term 0.
I20260809 05:24:35.334702 12666 raft_consensus.cc:2454] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9260b59fdb9487b8d39b5605082af51 in term 0.
I20260809 05:24:35.335312 12184 leader_election.cc:302] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4535f096c67c443a95389871e3551b79, 5948aff759944d1a9091325101d496b4, a9260b59fdb9487b8d39b5605082af51; no voters:
I20260809 05:24:35.335724 12770 raft_consensus.cc:2781] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:24:35.335857 12770 raft_consensus.cc:482] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:24:35.335955 12770 raft_consensus.cc:3037] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:35.338295 12770 raft_consensus.cc:504] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.338948 12770 leader_election.cc:288] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [CANDIDATE]: Term 1 election: Requested vote from peers 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637), 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097)
I20260809 05:24:35.339466 12666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83878663871141f7b2bde397d38d8729" candidate_uuid: "a9260b59fdb9487b8d39b5605082af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5948aff759944d1a9091325101d496b4"
I20260809 05:24:35.339605 12469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83878663871141f7b2bde397d38d8729" candidate_uuid: "a9260b59fdb9487b8d39b5605082af51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4535f096c67c443a95389871e3551b79"
I20260809 05:24:35.339689 12666 raft_consensus.cc:3037] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:35.339839 12469 raft_consensus.cc:3037] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:35.342964 12666 raft_consensus.cc:2454] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9260b59fdb9487b8d39b5605082af51 in term 1.
I20260809 05:24:35.342955 12469 raft_consensus.cc:2454] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9260b59fdb9487b8d39b5605082af51 in term 1.
I20260809 05:24:35.343396 12185 leader_election.cc:302] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5948aff759944d1a9091325101d496b4, a9260b59fdb9487b8d39b5605082af51; no voters:
I20260809 05:24:35.343762 12770 raft_consensus.cc:2781] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:35.344290 12770 raft_consensus.cc:686] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 1 LEADER]: Becoming Leader. State: Replica: a9260b59fdb9487b8d39b5605082af51, State: Running, Role: LEADER
I20260809 05:24:35.344822 12770 consensus_queue.cc:237] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } }
I20260809 05:24:35.349640 12081 catalog_manager.cc:5495] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9260b59fdb9487b8d39b5605082af51 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "a9260b59fdb9487b8d39b5605082af51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9260b59fdb9487b8d39b5605082af51" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 33349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 } health_report { overall_health: UNKNOWN } } }
W20260809 05:24:35.364626 12337 tablet.cc:2369] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:35.414054 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.408537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.22.1.65:33349)
0809 05:24:35.408743 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.408748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.408766 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:24:35.408952 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.408957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.409050 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.409211 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:24:35.409591 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.410285 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.411950 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.412364 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.412456 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.412937 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:24:35.413501 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:24:35.413617 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.413728 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.413799 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:24:35.422392 12469 raft_consensus.cc:1261] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Refusing update from remote peer a9260b59fdb9487b8d39b5605082af51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:35.422392 12666 raft_consensus.cc:1261] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Refusing update from remote peer a9260b59fdb9487b8d39b5605082af51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:35.423396 12773 consensus_queue.cc:1035] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4535f096c67c443a95389871e3551b79" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:35.423740 12770 consensus_queue.cc:1035] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5948aff759944d1a9091325101d496b4" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 41637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:35.450785 12085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:35.451102 12085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:35.455019 12085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Sending DeleteTablet for 3 replicas of tablet 83878663871141f7b2bde397d38d8729
I20260809 05:24:35.456308 12254 tablet_service.cc:1514] Processing DeleteTablet for tablet 83878663871141f7b2bde397d38d8729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:50928
I20260809 05:24:35.456416 12443 tablet_service.cc:1514] Processing DeleteTablet for tablet 83878663871141f7b2bde397d38d8729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:57306
I20260809 05:24:35.456670 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 12136
I20260809 05:24:35.457046 12638 tablet_service.cc:1514] Processing DeleteTablet for tablet 83878663871141f7b2bde397d38d8729 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:35 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:59688
I20260809 05:24:35.457501 12800 tablet_replica.cc:330] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: stopping tablet replica
I20260809 05:24:35.458025 12800 raft_consensus.cc:2229] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:35.458611 12800 raft_consensus.cc:2258] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:35.464831 12801 tablet_replica.cc:330] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: stopping tablet replica
I20260809 05:24:35.465346 12800 ts_tablet_manager.cc:1905] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:35.465425 12801 raft_consensus.cc:2229] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:35.466141 12801 raft_consensus.cc:2258] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:35.466773 12797 tablet_replica.cc:330] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: stopping tablet replica
I20260809 05:24:35.468372 12797 raft_consensus.cc:2229] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:24:35.471340 12797 raft_consensus.cc:2258] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:35.472996 12800 ts_tablet_manager.cc:1918] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:35.473194 12800 log.cc:1196] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/83878663871141f7b2bde397d38d8729
I20260809 05:24:35.473862 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.457844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.22.1.65:33349)
0809 05:24:35.458013 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.458018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.458038 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:24:35.458185 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.458191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.458254 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.458349 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:35.460067 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.461423 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.463315 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.463993 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.464117 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.467655 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:35.467668 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:35.467688 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:35.467734 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:35.470701 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.471603 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.471612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.471614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.471755 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.472183 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.472188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.472191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.472583 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:35.472888 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.473248 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.473489 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260809 05:24:35.473905 12800 ts_tablet_manager.cc:1939] T 83878663871141f7b2bde397d38d8729 P 4535f096c67c443a95389871e3551b79: Deleting consensus metadata
I20260809 05:24:35.475001 12330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:35.475685 12801 ts_tablet_manager.cc:1905] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:35.475946 12063 catalog_manager.cc:4841] TS 4535f096c67c443a95389871e3551b79 (127.22.1.66:36097): tablet 83878663871141f7b2bde397d38d8729 (table test-table [id=543ee424ff59407ab927e258972451af]) successfully deleted
I20260809 05:24:35.481907 12801 ts_tablet_manager.cc:1918] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:35.482064 12801 log.cc:1196] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/83878663871141f7b2bde397d38d8729
I20260809 05:24:35.482496 12801 ts_tablet_manager.cc:1939] T 83878663871141f7b2bde397d38d8729 P 5948aff759944d1a9091325101d496b4: Deleting consensus metadata
I20260809 05:24:35.484138 12064 catalog_manager.cc:4841] TS 5948aff759944d1a9091325101d496b4 (127.22.1.67:41637): tablet 83878663871141f7b2bde397d38d8729 (table test-table [id=543ee424ff59407ab927e258972451af]) successfully deleted
I20260809 05:24:35.475682 12797 ts_tablet_manager.cc:1905] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:35.577356 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 12136
W20260809 05:24:35.600795 12064 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:33349 (error 108)
I20260809 05:24:35.601590 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 12341
I20260809 05:24:35.602509 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.602035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50962)
0809 05:24:35.602210 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.602335 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 05:24:35.602751 12064 catalog_manager.cc:4584] TS a9260b59fdb9487b8d39b5605082af51 (127.22.1.65:33349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83878663871141f7b2bde397d38d8729: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
I20260809 05:24:35.615836 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.602444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.22.1.66:36097)
0809 05:24:35.602618 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.602625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.602645 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:24:35.602738 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.602744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.602818 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.602915 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:35.604257 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.605134 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.606686 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.607105 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.607217 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.609973 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:35.609988 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:35.610006 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:35.610040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:35.612912 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.613732 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.613740 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.613743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.613872 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.614257 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.614261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.614263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.614658 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:35.614938 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.615273 (+ 335us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.615500 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:24:35.616629 12534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:35.656412 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.655944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50970)
0809 05:24:35.656162 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.656269 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 05:24:35.657510 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.657193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50974)
0809 05:24:35.657310 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.657385 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260809 05:24:35.706164 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 12341
I20260809 05:24:35.718000 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.717541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50988)
0809 05:24:35.717745 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.717833 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 05:24:35.718976 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.718677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:50996)
0809 05:24:35.718795 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.718866 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260809 05:24:35.732347 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 12543
I20260809 05:24:35.745529 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.733087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.22.1.67:41637)
0809 05:24:35.733277 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:35.733282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:35.733321 (+ 39us) server_negotiation.cc:407] Connection header received
0809 05:24:35.733503 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:35.733507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:35.733602 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:35.733681 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:35.735071 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.735796 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.737140 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:35.737606 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:35.737698 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:35.740198 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:35.740214 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:35.740232 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:35.740284 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:35.743030 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.743760 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.743768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.743770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.743917 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:35.744212 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:35.744215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:35.744217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:35.744619 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:35.744837 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:35.745048 (+ 211us) server_negotiation.cc:299] Negotiation successful
0809 05:24:35.745263 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:24:35.746160 12730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:35.799731 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.799106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:51008)
0809 05:24:35.799363 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.799489 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260809 05:24:35.800930 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.800474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:51020)
0809 05:24:35.800694 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.800794 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 05:24:35.835583 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 12543
I20260809 05:24:35.859632 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 12038
I20260809 05:24:35.860857 12126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:35.974428 12758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:35.973790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:33349 (local address 127.0.0.1:51026)
0809 05:24:35.974002 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:35.974106 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:33349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 05:24:35.974457 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 12038
I20260809 05:24:36.004102 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--webserver_interface=127.22.1.126
--webserver_port=44691
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:36.337376 12811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:36.337697 12811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:36.337790 12811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:36.345125 12811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:36.345283 12811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:36.345337 12811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:36.345395 12811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:36.345440 12811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:36.355374 12811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=44691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:36.356755 12811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:36.358071 12811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:36.369537 12823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:36.369586 12824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:36.369505 12827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:36.370692 12811 server_base.cc:1034] running on GCE node
I20260809 05:24:36.371436 12811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:36.372821 12811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:36.374059 12811 hybrid_clock.cc:648] HybridClock initialized: now 1786253076373994 us; error 49 us; skew 500 ppm
I20260809 05:24:36.374404 12811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:36 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:36.377830 12811 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:36.380581 12811 webserver.cc:460] Webserver started at http://127.22.1.126:44691/ using document root <none> and password file <none>
I20260809 05:24:36.381474 12811 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:36.381603 12811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:36.387065 12811 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 05:24:36.389859 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:36.391153 12811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:36.391314 12811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "3c8bdf3527a243828957d46269780406"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "1262517ee6b3cd91889563cd9621f700"
server_key_iv: "cd46f5013f8c40430de9994b3130ec34"
server_key_version: "encryptionkey@0"
I20260809 05:24:36.392246 12811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:36.413856 12811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:36.418184 12811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:36.418440 12811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:36.436023 12811 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35749
I20260809 05:24:36.436087 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35749 every 8 connection(s)
I20260809 05:24:36.437602 12811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:36.441272 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12811
I20260809 05:24:36.442343 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:33349
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=37299
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:36.446180 12920 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:24:36.450697 12920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap starting.
I20260809 05:24:36.471856 12920 log.cc:824] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:36.488466 12920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:24:36.488986 12920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406: Bootstrap complete.
I20260809 05:24:36.498795 12920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:36.499727 12920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8bdf3527a243828957d46269780406, State: Initialized, Role: FOLLOWER
I20260809 05:24:36.500346 12920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:36.500581 12920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:36.500699 12920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:36.500861 12920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 2 FOLLOWER]: Advancing to term 3
I20260809 05:24:36.504822 12920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:36.505355 12920 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8bdf3527a243828957d46269780406; no voters:
I20260809 05:24:36.505996 12920 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 05:24:36.506146 12926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 3 FOLLOWER]: Leader election won for term 3
I20260809 05:24:36.507211 12926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [term 3 LEADER]: Becoming Leader. State: Replica: 3c8bdf3527a243828957d46269780406, State: Running, Role: LEADER
I20260809 05:24:36.507761 12926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } }
I20260809 05:24:36.508184 12920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:36.510202 12929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8bdf3527a243828957d46269780406. Latest consensus state: current_term: 3 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:36.510593 12929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:36.511121 12928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3c8bdf3527a243828957d46269780406" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8bdf3527a243828957d46269780406" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35749 } } }
I20260809 05:24:36.511389 12928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8bdf3527a243828957d46269780406 [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:36.514779 12939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:36.525609 12939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:36.527186 12939 catalog_manager.cc:1259] Loaded cluster ID: a9905725d5134e36b39b230db67b8f1e
I20260809 05:24:36.527395 12939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260809 05:24:36.529500 12939 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb07eaed9ad google::LogMessage::Fail() at ??:0
@ 0x7fb07eaf088a google::LogMessage::SendToLog() at ??:0
@ 0x7fb07eaed4de google::LogMessage::Flush() at ??:0
@ 0x7fb07eaeea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb094520512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb09451d976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb0945a5610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb0945a5439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb081739345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb08173c91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb08173c799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb081717eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb0849f16db start_thread at ??:0
@ 0x7fb07d98761f clone at ??:0
W20260809 05:24:36.856536 12923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:36.856909 12923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:36.857076 12923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:36.865561 12923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:36.865684 12923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:36.865839 12923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:36.875895 12923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:33349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=37299
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:36.877352 12923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:36.878737 12923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:36.892112 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:36.892197 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:36.892143 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:36.893172 12923 server_base.cc:1034] running on GCE node
I20260809 05:24:36.893826 12923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:36.894795 12923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:36.896014 12923 hybrid_clock.cc:648] HybridClock initialized: now 1786253076895940 us; error 55 us; skew 500 ppm
I20260809 05:24:36.896340 12923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:36 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253076, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:36.899574 12923 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:36.901185 12923 webserver.cc:460] Webserver started at http://127.22.1.65:37299/ using document root <none> and password file <none>
I20260809 05:24:36.901922 12923 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:36.902060 12923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:36.907763 12923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 05:24:36.910905 12994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:36.912370 12923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:24:36.912536 12923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "a9260b59fdb9487b8d39b5605082af51"
format_stamp: "Formatted at 2026-08-09 05:24:30 on dist-test-slave-rmhg"
server_key: "4835b7ceb2ec78124869af9eb30b4b08"
server_key_iv: "e9e54f3d1432cd56839b6cc7914531ef"
server_key_version: "encryptionkey@0"
I20260809 05:24:36.913648 12923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:36.955605 12923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:36.960013 12923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:36.960285 12923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:36.961385 12923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:36.963280 13006 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260809 05:24:36.968837 13006 ts_tablet_manager.cc:1852] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260809 05:24:36.969841 13006 ts_tablet_manager.cc:1905] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:36.977077 13006 ts_tablet_manager.cc:1918] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:36.977300 13006 log.cc:1196] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/83878663871141f7b2bde397d38d8729
I20260809 05:24:36.977739 13006 ts_tablet_manager.cc:1939] T 83878663871141f7b2bde397d38d8729 P a9260b59fdb9487b8d39b5605082af51: Deleting consensus metadata
I20260809 05:24:36.978103 12923 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260809 05:24:36.978195 12923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260809 05:24:36.978292 12923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:36.978348 12923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.009009 12923 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:33349
I20260809 05:24:37.009090 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:33349 every 8 connection(s)
I20260809 05:24:37.010500 12923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:37.015538 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.013566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:43963)
0809 05:24:37.014361 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.014620 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":300,"threads_started":1}
W20260809 05:24:37.016217 13171 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:35749 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:35749: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
I20260809 05:24:37.017032 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.016722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:59805)
0809 05:24:37.016858 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.016928 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260809 05:24:37.017881 13173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.017586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:54505)
0809 05:24:37.017693 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.017767 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260809 05:24:37.018126 13171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 05:24:37.019238 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 12923
I20260809 05:24:37.019870 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:36097
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=46325
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:37.360009 13174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:37.360322 13174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:37.360450 13174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:37.367992 13174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:37.368120 13174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:37.368275 13174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:37.378315 13174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:36097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=46325
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:37.379696 13174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:37.381071 13174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:37.394285 13184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:37.394258 13188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:37.394225 13186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:37.394984 13174 server_base.cc:1034] running on GCE node
I20260809 05:24:37.395660 13174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:37.396659 13174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:37.397862 13174 hybrid_clock.cc:648] HybridClock initialized: now 1786253077397808 us; error 50 us; skew 500 ppm
I20260809 05:24:37.398173 13174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:37 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:37.401351 13174 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:37.402906 13174 webserver.cc:460] Webserver started at http://127.22.1.66:46325/ using document root <none> and password file <none>
I20260809 05:24:37.403597 13174 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:37.403707 13174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:37.409231 13174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:24:37.412042 13197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.413576 13174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:37.413745 13174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "4535f096c67c443a95389871e3551b79"
format_stamp: "Formatted at 2026-08-09 05:24:31 on dist-test-slave-rmhg"
server_key: "42ad633bf7e8675008e667f4a7023197"
server_key_iv: "4a9841e3c49e11f921aaa1ffe1337a4c"
server_key_version: "encryptionkey@0"
I20260809 05:24:37.414830 13174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:37.434355 13174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:37.438661 13174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:37.438923 13174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:37.439965 13174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:37.441367 13174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:37.441457 13174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.441582 13174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:37.441653 13174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.473505 13174 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:36097
I20260809 05:24:37.473578 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:36097 every 8 connection(s)
I20260809 05:24:37.475065 13174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:37.477784 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 13174
I20260809 05:24:37.478552 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:41637
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:35749
--webserver_port=36703
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:24:37.480552 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.478130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:46327)
0809 05:24:37.479023 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.479273 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":309,"threads_started":1}
W20260809 05:24:37.481400 13366 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:35749 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:35749: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
I20260809 05:24:37.482352 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.481957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:45701)
0809 05:24:37.482138 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.482224 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 05:24:37.483253 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.482908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:57909)
0809 05:24:37.483039 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.483125 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260809 05:24:37.483486 13366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 05:24:37.829792 13369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:37.830096 13369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:37.830237 13369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:37.837702 13369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:37.837828 13369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:37.837971 13369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:37.848268 13369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:41637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=36703
--tserver_master_addrs=127.22.1.126:35749
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:37.849735 13369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:37.851131 13369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:37.865821 13379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:37.866284 13376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:37.866333 13377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:37.866668 13369 server_base.cc:1034] running on GCE node
I20260809 05:24:37.867329 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:37.868309 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:37.869508 13369 hybrid_clock.cc:648] HybridClock initialized: now 1786253077869455 us; error 40 us; skew 500 ppm
I20260809 05:24:37.869830 13369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:37 dist-test-slave-rmhg krb5kdc[11224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253077, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:37.873004 13369 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:37.874517 13369 webserver.cc:460] Webserver started at http://127.22.1.67:36703/ using document root <none> and password file <none>
I20260809 05:24:37.875236 13369 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:37.875360 13369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:37.880831 13369 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260809 05:24:37.883635 13390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.884999 13369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:24:37.885200 13369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "5948aff759944d1a9091325101d496b4"
format_stamp: "Formatted at 2026-08-09 05:24:32 on dist-test-slave-rmhg"
server_key: "433588f0abe330696cc15c5e3abef2ab"
server_key_iv: "d0f8a296f9d4a29c2444200b3c9d6442"
server_key_version: "encryptionkey@0"
I20260809 05:24:37.886209 13369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:37.907099 13369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:37.911592 13369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:37.911856 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:37.912868 13369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:37.914307 13369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:37.914407 13369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.914516 13369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:37.914597 13369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:37.945683 13369 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:41637
I20260809 05:24:37.945752 13558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:41637 every 8 connection(s)
I20260809 05:24:37.947141 13369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:37.952199 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.950304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:38865)
0809 05:24:37.951090 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.951293 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":274,"threads_started":1}
W20260809 05:24:37.953002 13560 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:35749 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:35749: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
I20260809 05:24:37.953895 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.953556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:40315)
0809 05:24:37.953703 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.953777 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260809 05:24:37.954897 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:37.954548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:41609)
0809 05:24:37.954701 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:37.954782 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 05:24:37.955216 13560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 05:24:37.956617 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 13369
I20260809 05:24:38.019613 13566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:38.018752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:39055)
0809 05:24:38.019263 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:38.019363 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":262,"threads_started":1}
I20260809 05:24:38.485057 13567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:38.484170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:47495)
0809 05:24:38.484705 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:38.484827 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":260,"threads_started":1}
I20260809 05:24:38.956650 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:38.955767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:34743)
0809 05:24:38.956292 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:38.956412 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":271,"threads_started":1}
I20260809 05:24:39.021225 13570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:39.020501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:58261)
0809 05:24:39.020942 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:39.021028 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":235,"threads_started":1}
I20260809 05:24:39.486783 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:39.486027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:38203)
0809 05:24:39.486518 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:39.486618 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":218,"threads_started":1}
I20260809 05:24:39.958398 13572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:39.957663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:43503)
0809 05:24:39.958137 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:39.958225 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":233,"threads_started":1}
I20260809 05:24:40.022878 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:40.022108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:38935)
0809 05:24:40.022585 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:40.022680 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20260809 05:24:40.488546 13577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:40.487766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:45553)
0809 05:24:40.488265 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:40.488358 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260809 05:24:40.960011 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:40.959281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:58951)
0809 05:24:40.959732 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:40.959826 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260809 05:24:41.024487 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:41.023749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:47565)
0809 05:24:41.024221 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:41.024308 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260809 05:24:41.490384 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:41.489540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:47739)
0809 05:24:41.490065 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:41.490157 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":265,"threads_started":1}
I20260809 05:24:41.961817 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:41.960981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:58627)
0809 05:24:41.961524 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:41.961617 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
I20260809 05:24:42.026124 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:42.025380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:33741)
0809 05:24:42.025841 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:42.025924 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":229,"threads_started":1}
I20260809 05:24:42.492091 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:42.491307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:54443)
0809 05:24:42.491808 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:42.491896 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":263,"threads_started":1}
I20260809 05:24:42.963683 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:42.962839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:40723)
0809 05:24:42.963398 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:42.963496 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":315,"threads_started":1}
I20260809 05:24:43.027630 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:43.026937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:35559)
0809 05:24:43.027376 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:43.027460 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260809 05:24:43.493798 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:43.492984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:54033)
0809 05:24:43.493507 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:43.493608 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":269,"threads_started":1}
I20260809 05:24:43.965598 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:43.964678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:46557)
0809 05:24:43.965231 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:43.965361 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":295,"threads_started":1}
I20260809 05:24:44.029394 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:44.028553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:33705)
0809 05:24:44.029056 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:44.029143 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":225,"threads_started":1}
I20260809 05:24:44.495689 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:44.494794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:34709)
0809 05:24:44.495371 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:44.495462 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":269,"threads_started":1}
I20260809 05:24:44.967581 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:44.966598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:54827)
0809 05:24:44.967161 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:44.967252 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":244,"threads_started":1}
I20260809 05:24:45.030992 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:45.030299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:35365)
0809 05:24:45.030741 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:45.030822 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
I20260809 05:24:45.497386 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:45.496632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:49335)
0809 05:24:45.497104 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:45.497217 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":227,"threads_started":1}
I20260809 05:24:45.969367 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:45.968543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:46547)
0809 05:24:45.969047 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:45.969138 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260809 05:24:46.032704 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:46.031915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:42399)
0809 05:24:46.032416 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:46.032534 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260809 05:24:46.499174 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:46.498357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:51397)
0809 05:24:46.498896 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:46.498991 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":287,"threads_started":1}
I20260809 05:24:46.971103 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:46.970242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:34285)
0809 05:24:46.970805 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:46.970902 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
I20260809 05:24:47.034489 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:47.033768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:58805)
0809 05:24:47.034226 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:47.034316 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":230,"threads_started":1}
I20260809 05:24:47.500865 13614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:47.500134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:59001)
0809 05:24:47.500605 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:47.500692 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260809 05:24:47.972930 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:47.972133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:46257)
0809 05:24:47.972669 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:47.972758 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":272,"threads_started":1}
I20260809 05:24:48.036087 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:48.035338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:58439)
0809 05:24:48.035818 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:48.035901 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":249,"threads_started":1}
I20260809 05:24:48.502666 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:48.501889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:44821)
0809 05:24:48.502391 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:48.502474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
I20260809 05:24:48.974665 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:48.973922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:41607)
0809 05:24:48.974383 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:48.974475 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260809 05:24:49.037832 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:49.037079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:39983)
0809 05:24:49.037581 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:49.037671 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":282,"threads_started":1}
I20260809 05:24:49.504333 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:49.503592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:52975)
0809 05:24:49.504077 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:49.504165 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":243,"threads_started":1}
I20260809 05:24:49.976267 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:49.975551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:46871)
0809 05:24:49.976026 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:49.976108 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
I20260809 05:24:50.039669 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:50.038878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:38181)
0809 05:24:50.039355 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:50.039465 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":227,"threads_started":1}
I20260809 05:24:50.506093 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:50.505321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:47163)
0809 05:24:50.505784 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:50.505896 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":232,"threads_started":1}
I20260809 05:24:50.978006 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:50.977225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:43747)
0809 05:24:50.977722 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:50.977817 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":224,"threads_started":1}
I20260809 05:24:51.041360 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:51.040591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:41675)
0809 05:24:51.041043 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:51.041128 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":218,"threads_started":1}
I20260809 05:24:51.507876 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:51.507065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:35585)
0809 05:24:51.507583 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:51.507678 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":256,"threads_started":1}
I20260809 05:24:51.979790 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:51.978975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:40843)
0809 05:24:51.979505 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:51.979599 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":277,"threads_started":1}
I20260809 05:24:52.043190 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:52.042451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.65:59349)
0809 05:24:52.042906 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:52.042995 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":217,"threads_started":1}
I20260809 05:24:52.509742 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:52.508997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.66:49719)
0809 05:24:52.509460 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:52.509560 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":239,"threads_started":1}
I20260809 05:24:52.968869 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 12923
I20260809 05:24:52.982046 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:52.981088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35749 (local address 127.22.1.67:55343)
0809 05:24:52.981661 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:52.981786 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:35749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260809 05:24:52.989431 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:52.970262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.22.1.65:33349)
0809 05:24:52.970843 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:52.970864 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:52.970918 (+ 54us) server_negotiation.cc:407] Connection header received
0809 05:24:52.972617 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:52.972627 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:52.973088 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:52.973463 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:52.975378 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:52.976766 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:52.978286 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:52.979050 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:52.979215 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:52.982647 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:52.982657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:52.982675 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:52.982719 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:52.985830 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:52.986587 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:52.986595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:52.986603 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:52.986800 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:52.987209 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:52.987213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:52.987215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:52.987851 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:52.988112 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:52.988634 (+ 522us) server_negotiation.cc:299] Negotiation successful
0809 05:24:52.989010 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":352,"threads_started":1}
I20260809 05:24:52.991108 13165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:53.077584 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 12923
I20260809 05:24:53.104672 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 13174
I20260809 05:24:53.122490 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:53.105762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.22.1.66:36097)
0809 05:24:53.106340 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:53.106355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:53.106394 (+ 39us) server_negotiation.cc:407] Connection header received
0809 05:24:53.107976 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:53.107986 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:53.108432 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:53.108736 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:53.110251 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:53.111527 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:53.112784 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:53.113595 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:53.113736 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:53.116236 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:53.116243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:53.116269 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:53.116308 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:53.119200 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:53.120005 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:53.120024 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:53.120031 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:53.120208 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:53.120516 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:53.120520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:53.120523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:53.121079 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:53.121361 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:53.121776 (+ 415us) server_negotiation.cc:299] Negotiation successful
0809 05:24:53.122201 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":285,"threads_started":1}
I20260809 05:24:53.123903 13359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:53.207907 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 13174
I20260809 05:24:53.231316 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 13369
I20260809 05:24:53.250315 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:53.232283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.22.1.67:41637)
0809 05:24:53.232919 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:53.232934 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:53.232984 (+ 50us) server_negotiation.cc:407] Connection header received
0809 05:24:53.234710 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:53.234721 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:53.235203 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:53.235519 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:53.236841 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:53.238337 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:53.239718 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:53.240437 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:53.240583 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:53.243618 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:53.243628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:53.243647 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:53.243690 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:53.246909 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:53.247686 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:53.247694 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:53.247702 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:53.247901 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:53.248291 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:53.248294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:53.248296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:53.248910 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:53.249190 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:53.249540 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 05:24:53.249995 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":309,"threads_started":1}
I20260809 05:24:53.251820 13551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:53.337065 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 13369
2026-08-09T05:24:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24836 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:24:53 dist-test-slave-rmhg krb5kdc[13656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:24:53 dist-test-slave-rmhg krb5kdc[13656](info): set up 2 sockets
Aug 09 05:24:53 dist-test-slave-rmhg krb5kdc[13656](info): commencing operation
krb5kdc: starting...
W20260809 05:24:54.555554 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:24:54 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:24:54.844990 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:55.195394 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:55.195695 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:55.195827 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:55.203039 13676 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 05:24:55.203140 13676 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 05:24:55.203186 13676 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 05:24:55.203256 13676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:24:55.203320 13676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:55.203411 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:55.203467 13676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:24:55.203547 13676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:24:55.213531 13676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:55.214813 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:55.216161 13676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:55.228376 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:55.228387 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:55.228418 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:55.228740 13676 server_base.cc:1034] running on GCE node
I20260809 05:24:55.229875 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:55.231361 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:55.232600 13676 hybrid_clock.cc:648] HybridClock initialized: now 1786253095232550 us; error 50 us; skew 500 ppm
I20260809 05:24:55.232932 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:55 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253095, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:55.236004 13676 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:24:55.237491 13676 webserver.cc:460] Webserver started at http://127.22.1.126:41011/ using document root <none> and password file <none>
I20260809 05:24:55.238184 13676 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:55.238288 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:55.238603 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:55.241667 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "dd54a710d4f844019f0974de7832bb7d"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "497822f90548f5e136029da5c1b2012f"
server_key_iv: "79a291c570dfef535e5dbf8a8b2d900b"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.242425 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "dd54a710d4f844019f0974de7832bb7d"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "497822f90548f5e136029da5c1b2012f"
server_key_iv: "79a291c570dfef535e5dbf8a8b2d900b"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.247836 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:24:55.251721 13693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:55.253242 13676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 05:24:55.253424 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "dd54a710d4f844019f0974de7832bb7d"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "497822f90548f5e136029da5c1b2012f"
server_key_iv: "79a291c570dfef535e5dbf8a8b2d900b"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.253659 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:55.278697 13676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:55.282825 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:55.283092 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:55.300848 13676 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38883
I20260809 05:24:55.300869 13775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38883 every 8 connection(s)
I20260809 05:24:55.302253 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:24:55.304138 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 13676
I20260809 05:24:55.304330 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:24:55.304986 22533 external_mini_cluster.cc:1428] Setting key 635208d32f62dfcb1c28b78feb982b05
I20260809 05:24:55.306262 13776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:55.318313 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap starting.
Aug 09 05:24:55 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:55.321673 13776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:55.322757 13776 log.cc:824] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:55.325624 13776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: No bootstrap required, opened a new log
I20260809 05:24:55.326958 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:55.306848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51554 (local address 127.22.1.126:38883)
0809 05:24:55.307684 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:55.307701 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:55.308025 (+ 324us) server_negotiation.cc:407] Connection header received
0809 05:24:55.309796 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:55.309820 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:55.310210 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:55.310598 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:55.312106 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.313496 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.314829 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.315517 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.315717 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:55.320010 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:55.320018 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:55.320033 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:55.320085 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:55.323023 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:55.323692 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:55.323698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:55.323705 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:55.323876 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:55.324255 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:55.324258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:55.324260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:55.324780 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:55.325067 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:55.325544 (+ 477us) server_negotiation.cc:299] Negotiation successful
0809 05:24:55.325988 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":345,"threads_started":1}
I20260809 05:24:55.333127 13776 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:24:55.333600 13776 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:55.333698 13776 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Initialized, Role: FOLLOWER
I20260809 05:24:55.334218 13776 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:24:55.334453 13776 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:24:55.334582 13776 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:24:55.334738 13776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:55.337037 13776 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:24:55.337612 13776 leader_election.cc:302] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd54a710d4f844019f0974de7832bb7d; no voters:
I20260809 05:24:55.338230 13776 leader_election.cc:288] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:24:55.338385 13782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:55.339123 13782 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 LEADER]: Becoming Leader. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Running, Role: LEADER
I20260809 05:24:55.339718 13782 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:24:55.340862 13776 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:24:55.342563 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd54a710d4f844019f0974de7832bb7d. Latest consensus state: current_term: 1 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:24:55.342564 13783 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:24:55.343016 13783 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:55.343591 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:24:55.344673 13788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:24:55.349392 13788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:24:55.358175 13788 catalog_manager.cc:1347] Generated new cluster ID: 5a0badd7cc624474843a21caf2f6b22a
I20260809 05:24:55.358395 13788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:24:55.385015 13788 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:24:55.386381 13788 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:24:55.396962 13788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Generated new TSK 0
I20260809 05:24:55.397858 13788 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:24:55.464890 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:55.816231 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:55.816552 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:55.816653 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:55.824244 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:55.824363 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:55.824532 13809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:24:55.834936 13809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:55.836208 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:55.837594 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:55.851023 13817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:55.850973 13818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:55.850973 13820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:55.851809 13809 server_base.cc:1034] running on GCE node
I20260809 05:24:55.852370 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:55.853459 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:55.854681 13809 hybrid_clock.cc:648] HybridClock initialized: now 1786253095854635 us; error 34 us; skew 500 ppm
I20260809 05:24:55.855006 13809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:55 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253095, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:55.858619 13809 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:24:55.860304 13809 webserver.cc:460] Webserver started at http://127.22.1.65:37801/ using document root <none> and password file <none>
I20260809 05:24:55.861054 13809 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:55.861194 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:55.861523 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:55.864550 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "981d9ef40a654fe283363b75c5db3898"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "4f634c021fc5321003e9c8f0f0f082fc"
server_key_iv: "7744a9639105dcfdb70743368fed0aa3"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.865348 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "981d9ef40a654fe283363b75c5db3898"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "4f634c021fc5321003e9c8f0f0f082fc"
server_key_iv: "7744a9639105dcfdb70743368fed0aa3"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.870606 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:24:55.874147 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:55.875465 13809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:55.875648 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "981d9ef40a654fe283363b75c5db3898"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "4f634c021fc5321003e9c8f0f0f082fc"
server_key_iv: "7744a9639105dcfdb70743368fed0aa3"
server_key_version: "encryptionkey@0"
I20260809 05:24:55.875826 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:55.893548 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:55.897832 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:55.898124 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:55.899196 13809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:55.900702 13809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:55.900801 13809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:55.900928 13809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:55.900982 13809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:55.933702 13809 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:37507
I20260809 05:24:55.933769 14003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:37507 every 8 connection(s)
I20260809 05:24:55.935034 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:24:55.944788 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 13809
I20260809 05:24:55.945017 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:24:55.945808 22533 external_mini_cluster.cc:1428] Setting key 6549662835ef183a29c3e2dadadaa8d6
Aug 09 05:24:55 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253095, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:55.955885 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:55.937760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:40909 (local address 127.22.1.126:38883)
0809 05:24:55.938229 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:55.938236 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:55.939282 (+ 1046us) server_negotiation.cc:407] Connection header received
0809 05:24:55.941217 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:55.941222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:55.941295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:55.941392 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:55.942979 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.943719 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.945485 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.946110 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.946246 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:55.950261 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:55.950267 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:55.950270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:55.950304 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:55.952688 (+ 2384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:55.953561 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:55.953565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:55.953567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:55.953688 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:55.954130 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:55.954133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:55.954135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:55.954399 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:55.954519 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:55.955414 (+ 895us) server_negotiation.cc:299] Negotiation successful
0809 05:24:55.955605 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
I20260809 05:24:55.957135 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:55.938336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:40909)
0809 05:24:55.939093 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:55.939147 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:55.940906 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:55.941634 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:55.941643 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:55.942230 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:55.942829 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:55.942846 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.943877 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.943885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:55.945319 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:55.945333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:55.946337 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:55.946342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:55.946612 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:55.947298 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:55.947336 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:55.950024 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:55.952827 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:55.952835 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:55.952853 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:55.953450 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:55.953787 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:55.953791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:55.953794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:55.954040 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:55.954524 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:55.954532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:55.954965 (+ 433us) client_negotiation.cc:769] Sending connection context
0809 05:24:55.955268 (+ 303us) client_negotiation.cc:240] Negotiation successful
0809 05:24:55.955791 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":287,"threads_started":1}
I20260809 05:24:55.958767 14005 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:24:55.959084 14005 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:55.959811 14005 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:24:55.961894 13720 ts_manager.cc:194] Registered new tserver with Master: 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507)
I20260809 05:24:55.963335 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:40909
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:24:56.006217 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:56.355530 14013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:56.355818 14013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:56.355911 14013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:56.363270 14013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:56.363387 14013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:56.363514 14013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:24:56.373245 14013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:56.374648 14013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:56.376367 14013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:56.391326 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:56.391372 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:56.391515 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:56.392146 14013 server_base.cc:1034] running on GCE node
I20260809 05:24:56.392745 14013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:56.393733 14013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:56.394930 14013 hybrid_clock.cc:648] HybridClock initialized: now 1786253096394875 us; error 48 us; skew 500 ppm
I20260809 05:24:56.395239 14013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:56 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253096, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:56.398288 14013 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:24:56.399751 14013 webserver.cc:460] Webserver started at http://127.22.1.66:37333/ using document root <none> and password file <none>
I20260809 05:24:56.400411 14013 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:56.400542 14013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:56.400833 14013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:56.403805 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "0aa57b5f27064426afb0bc9e11915869"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "4a1f19d4ac3c425c51cda8e153f7b08b"
server_key_iv: "70586e092101030c29d918c3d4246f63"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.404569 14013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "0aa57b5f27064426afb0bc9e11915869"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "4a1f19d4ac3c425c51cda8e153f7b08b"
server_key_iv: "70586e092101030c29d918c3d4246f63"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.409615 14013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260809 05:24:56.412959 14034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:56.414317 14013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:56.414484 14013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "0aa57b5f27064426afb0bc9e11915869"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "4a1f19d4ac3c425c51cda8e153f7b08b"
server_key_iv: "70586e092101030c29d918c3d4246f63"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.414673 14013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:56.436282 14013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:56.440495 14013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:56.440786 14013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:56.441901 14013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:56.443291 14013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:56.443377 14013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:56.443507 14013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:56.443562 14013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:56.475584 14013 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:46781
I20260809 05:24:56.475638 14212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:46781 every 8 connection(s)
I20260809 05:24:56.476950 14013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:24:56.484864 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 14013
I20260809 05:24:56.485095 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:24:56.485687 22533 external_mini_cluster.cc:1428] Setting key 603533fe861668767be782cb79dd9aa1
Aug 09 05:24:56 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253096, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:56.498994 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:56.479882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:56473 (local address 127.22.1.126:38883)
0809 05:24:56.480388 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:56.480393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:56.481437 (+ 1044us) server_negotiation.cc:407] Connection header received
0809 05:24:56.483421 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:56.483427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:56.483509 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:56.483606 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:56.485573 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:56.486427 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:56.488442 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:56.488890 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:56.488984 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:56.493086 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:56.493091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:56.493094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:56.493129 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:56.495621 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:56.496521 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:56.496525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:56.496527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:56.496663 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:56.497173 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:56.497179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:56.497182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:56.497499 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:56.497630 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:56.498519 (+ 889us) server_negotiation.cc:299] Negotiation successful
0809 05:24:56.498721 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260809 05:24:56.500253 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:56.480510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:56473)
0809 05:24:56.481282 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:56.481322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:56.483109 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:56.483867 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:56.483875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:56.484446 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:56.485338 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:56.485365 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:56.486589 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:56.486594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:56.488277 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:56.488293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:56.489136 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:56.489144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:56.489476 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:56.490180 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:56.490214 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:56.492826 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:56.495770 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:56.495779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:56.495799 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:56.496396 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:56.496759 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:56.496763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:56.496766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:56.497025 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:56.497621 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:56.497629 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:56.498107 (+ 478us) client_negotiation.cc:769] Sending connection context
0809 05:24:56.498429 (+ 322us) client_negotiation.cc:240] Negotiation successful
0809 05:24:56.499034 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":280,"threads_started":1}
I20260809 05:24:56.501865 14213 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:24:56.502202 14213 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:56.503126 14213 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:24:56.504777 13720 ts_manager.cc:194] Registered new tserver with Master: 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
I20260809 05:24:56.505656 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:56473
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:24:56.550287 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:24:56.891376 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:24:56.891650 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:24:56.891755 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:24:56.898933 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:24:56.899075 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:24:56.899199 14225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:24:56.909183 14225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:24:56.910524 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:24:56.911859 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:24:56.926047 14233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:56.926002 14236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:24:56.926473 14232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:24:56.926964 14225 server_base.cc:1034] running on GCE node
I20260809 05:24:56.927517 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:24:56.928445 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:24:56.929656 14225 hybrid_clock.cc:648] HybridClock initialized: now 1786253096929604 us; error 38 us; skew 500 ppm
I20260809 05:24:56.929979 14225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:24:56 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253096, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:24:56.933030 14225 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:24:56.934588 14225 webserver.cc:460] Webserver started at http://127.22.1.67:41555/ using document root <none> and password file <none>
I20260809 05:24:56.935271 14225 fs_manager.cc:359] Metadata directory not provided
I20260809 05:24:56.935393 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:24:56.935700 14225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:24:56.938779 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "a7bab1fa28d44acea582accfe6177f67"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "23797552404c17cc5caae0ac0078d52e"
server_key_iv: "925f99979d90d762939aef225de132bd"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.939591 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "a7bab1fa28d44acea582accfe6177f67"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "23797552404c17cc5caae0ac0078d52e"
server_key_iv: "925f99979d90d762939aef225de132bd"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.944993 14225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260809 05:24:56.948554 14246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:56.949939 14225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:24:56.950086 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "a7bab1fa28d44acea582accfe6177f67"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "23797552404c17cc5caae0ac0078d52e"
server_key_iv: "925f99979d90d762939aef225de132bd"
server_key_version: "encryptionkey@0"
I20260809 05:24:56.950276 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:24:56.966287 14005 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
I20260809 05:24:56.971374 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:24:56.975620 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:24:56.975870 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:24:56.976866 14225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:24:56.978288 14225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:24:56.978372 14225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:56.978503 14225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:24:56.978596 14225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:24:57.010241 14225 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:44011
I20260809 05:24:57.010293 14405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:44011 every 8 connection(s)
I20260809 05:24:57.011547 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:24:57.017998 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 14225
I20260809 05:24:57.018178 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:24:57.018644 22533 external_mini_cluster.cc:1428] Setting key 09535f786a663de67680ca862a52ff04
Aug 09 05:24:57 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253096, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:24:57.031602 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.014115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:59783 (local address 127.22.1.126:38883)
0809 05:24:57.014563 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.014567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.015455 (+ 888us) server_negotiation.cc:407] Connection header received
0809 05:24:57.017393 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.017399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.017490 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.017591 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:57.019669 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.020562 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.022206 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.022610 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.022720 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.026231 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:57.026236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:57.026240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:57.026276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:57.028856 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.029628 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.029632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.029634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.029766 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.030111 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.030114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.030116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.030377 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:57.030500 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.031145 (+ 645us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.031316 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":220,"threads_started":1}
I20260809 05:24:57.032632 14413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.014576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:59783)
0809 05:24:57.015291 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.015334 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.017062 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.017887 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.017897 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.018637 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:24:57.019477 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.019503 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.020722 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.020726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.022067 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.022080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.022835 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.022842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.023123 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.023739 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:24:57.023766 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:24:57.025992 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:24:57.028999 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:57.029007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:57.029021 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:57.029517 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:57.029844 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:24:57.029847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:24:57.029849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:24:57.030028 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:24:57.030495 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:24:57.030501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:24:57.030833 (+ 332us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.031068 (+ 235us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.031473 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":279,"threads_started":1}
I20260809 05:24:57.034188 14406 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:24:57.034519 14406 heartbeater.cc:464] Registering TS with master...
I20260809 05:24:57.035239 14406 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:24:57.036617 13720 ts_manager.cc:194] Registered new tserver with Master: a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011)
I20260809 05:24:57.037493 13720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:59783
I20260809 05:24:57.047008 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:24:57.065122 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.051592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.22.1.126:38883)
0809 05:24:57.051749 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.051753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.052191 (+ 438us) server_negotiation.cc:407] Connection header received
0809 05:24:57.052359 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.052365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.052438 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.052518 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:57.054153 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.054908 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.056266 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.056685 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.056795 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.059599 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:57.059605 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:57.059609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:57.059648 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:57.062296 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.063078 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.063083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.063085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.063231 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.063615 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.063621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.063623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.063971 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:57.064201 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.064568 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.064798 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 05:24:57.069311 13720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:24:57.076211 13720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:24:57.103705 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.089296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.22.1.65:37507)
0809 05:24:57.090115 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.090125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.093775 (+ 3650us) server_negotiation.cc:407] Connection header received
0809 05:24:57.094113 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.094122 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.094430 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.094617 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:57.095706 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.097529 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.100727 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.102420 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.102561 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.102812 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.103135 (+ 323us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.103355 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":496,"threads_started":1}
I20260809 05:24:57.104316 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.088972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:37950)
0809 05:24:57.093627 (+ 4655us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.093655 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.093894 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.095013 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.095021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.095055 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:57.095503 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.095515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.097766 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.097772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.100454 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.100478 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.102631 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.102638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.102870 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.102932 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.103077 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.103296 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4513,"thread_start_us":4365,"threads_started":1}
I20260809 05:24:57.108026 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.088439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:46781 (local address 127.0.0.1:41640)
0809 05:24:57.089241 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.089291 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.089463 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.100778 (+ 11315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.100783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.100815 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:57.101266 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.101282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.103421 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.103426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.105750 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.105767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.107209 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.107219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.107402 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.107432 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.107527 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.107669 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":627,"threads_started":1}
I20260809 05:24:57.108274 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.088684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.22.1.66:46781)
0809 05:24:57.099995 (+ 11311us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.100004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.100037 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:24:57.100151 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.100160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.100508 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.101531 (+ 1023us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:57.101808 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.103253 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.105993 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.106974 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.107104 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.107327 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.107621 (+ 294us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.107804 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11115,"thread_start_us":11036,"threads_started":1}
I20260809 05:24:57.109906 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.089966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44011 (local address 127.0.0.1:58874)
0809 05:24:57.100526 (+ 10560us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.100550 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.100696 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.101797 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.101802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.101830 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:57.102144 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.102152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.105343 (+ 3191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.105348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.107339 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.107355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.109138 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.109144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.109336 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.109368 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.109456 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.109582 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10416,"thread_start_us":10236,"threads_started":1}
I20260809 05:24:57.109256 13901 tablet_service.cc:1467] Processing CreateTablet for tablet 824c89d43a474c5aa851f21f72672298 (DEFAULT_TABLE table=test-table [id=fbd819e673454b7991757fea1b7f8d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:57.110654 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.090374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.22.1.67:44011)
0809 05:24:57.099664 (+ 9290us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.099675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.100854 (+ 1179us) server_negotiation.cc:407] Connection header received
0809 05:24:57.100987 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.101000 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.101326 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.101539 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:57.102326 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.105092 (+ 2766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.107529 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.108948 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.109076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.109368 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.110145 (+ 777us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.110335 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9051,"thread_start_us":8859,"threads_started":1}
I20260809 05:24:57.110922 13901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c89d43a474c5aa851f21f72672298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:57.112202 14106 tablet_service.cc:1467] Processing CreateTablet for tablet 824c89d43a474c5aa851f21f72672298 (DEFAULT_TABLE table=test-table [id=fbd819e673454b7991757fea1b7f8d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:57.113660 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c89d43a474c5aa851f21f72672298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:57.113783 14324 tablet_service.cc:1467] Processing CreateTablet for tablet 824c89d43a474c5aa851f21f72672298 (DEFAULT_TABLE table=test-table [id=fbd819e673454b7991757fea1b7f8d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:24:57.115097 14324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824c89d43a474c5aa851f21f72672298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:24:57.126503 14440 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Bootstrap starting.
I20260809 05:24:57.129884 14445 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Bootstrap starting.
I20260809 05:24:57.130784 14440 tablet_bootstrap.cc:654] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:57.131358 14443 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Bootstrap starting.
I20260809 05:24:57.133054 14440 log.cc:824] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:57.133777 14445 tablet_bootstrap.cc:654] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:57.135039 14443 tablet_bootstrap.cc:654] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Neither blocks nor log segments found. Creating new log.
I20260809 05:24:57.135088 14445 log.cc:824] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:57.135912 14440 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: No bootstrap required, opened a new log
I20260809 05:24:57.136214 14440 ts_tablet_manager.cc:1397] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 05:24:57.136219 14443 log.cc:824] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Log is configured to *not* fsync() on all Append() calls
I20260809 05:24:57.138309 14445 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: No bootstrap required, opened a new log
I20260809 05:24:57.138643 14445 ts_tablet_manager.cc:1397] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260809 05:24:57.139426 14443 tablet_bootstrap.cc:492] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: No bootstrap required, opened a new log
I20260809 05:24:57.139712 14443 ts_tablet_manager.cc:1397] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 05:24:57.144907 14440 raft_consensus.cc:348] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.145337 14440 raft_consensus.cc:374] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:57.145483 14440 raft_consensus.cc:729] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981d9ef40a654fe283363b75c5db3898, State: Initialized, Role: FOLLOWER
I20260809 05:24:57.146090 14440 consensus_queue.cc:260] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.147732 14440 ts_tablet_manager.cc:1428] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260809 05:24:57.149955 14443 raft_consensus.cc:348] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.150403 14443 raft_consensus.cc:374] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:57.150148 14445 raft_consensus.cc:348] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.150497 14443 raft_consensus.cc:729] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bab1fa28d44acea582accfe6177f67, State: Initialized, Role: FOLLOWER
I20260809 05:24:57.150635 14445 raft_consensus.cc:374] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:24:57.150743 14445 raft_consensus.cc:729] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa57b5f27064426afb0bc9e11915869, State: Initialized, Role: FOLLOWER
I20260809 05:24:57.151185 14443 consensus_queue.cc:260] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.151352 14445 consensus_queue.cc:260] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.153802 14213 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
I20260809 05:24:57.154563 14445 ts_tablet_manager.cc:1428] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20260809 05:24:57.155419 14406 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
I20260809 05:24:57.156177 14443 ts_tablet_manager.cc:1428] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260809 05:24:57.187484 14006 tablet.cc:2369] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:57.203368 14452 raft_consensus.cc:482] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:24:57.203689 14452 raft_consensus.cc:504] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.205451 14452 leader_election.cc:288] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507)
I20260809 05:24:57.214807 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.205905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:46507 (local address 127.22.1.67:44011)
0809 05:24:57.206116 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.206122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.206835 (+ 713us) server_negotiation.cc:407] Connection header received
0809 05:24:57.207048 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.207052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.207125 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.207224 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:57.208049 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.209203 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.212341 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.213610 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.213746 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.213858 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.214389 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.214548 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 05:24:57.214821 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.206036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57223 (local address 127.22.1.65:37507)
0809 05:24:57.206236 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.206242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.206900 (+ 658us) server_negotiation.cc:407] Connection header received
0809 05:24:57.207039 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.207045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.207126 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.207225 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:24:57.208046 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.209203 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.212341 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.213620 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.213746 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.213859 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.214390 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.214566 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:24:57.215034 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.205697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44011 (local address 127.22.1.66:46507)
0809 05:24:57.206699 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.206725 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.206888 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.207267 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.207272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.207303 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:57.207876 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.207890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.209351 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.209357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.212132 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.212151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.213918 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.213926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.214175 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.214238 (+ 63us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.214338 (+ 100us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.214515 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":226,"thread_start_us":591,"threads_started":1}
I20260809 05:24:57.215054 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.205794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.22.1.66:57223)
0809 05:24:57.206795 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0809 05:24:57.206810 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:24:57.206905 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:24:57.207267 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:24:57.207272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:24:57.207371 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:24:57.207876 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.207890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.209352 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.209357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.212132 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:24:57.212151 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.213918 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.213926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:24:57.214180 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.214233 (+ 53us) client_negotiation.cc:769] Sending connection context
0809 05:24:57.214338 (+ 105us) client_negotiation.cc:240] Negotiation successful
0809 05:24:57.214535 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":196,"spinlock_wait_cycles":25856,"thread_start_us":598,"threads_started":1}
I20260809 05:24:57.216244 13934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c89d43a474c5aa851f21f72672298" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d9ef40a654fe283363b75c5db3898" is_pre_election: true
I20260809 05:24:57.216244 14349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c89d43a474c5aa851f21f72672298" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bab1fa28d44acea582accfe6177f67" is_pre_election: true
I20260809 05:24:57.216882 14349 raft_consensus.cc:2454] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 0.
I20260809 05:24:57.216882 13934 raft_consensus.cc:2454] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 0.
I20260809 05:24:57.217736 14040 leader_election.cc:302] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67; no voters:
I20260809 05:24:57.218182 14452 raft_consensus.cc:2781] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:24:57.218349 14452 raft_consensus.cc:482] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:24:57.218456 14452 raft_consensus.cc:3037] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:57.220878 14452 raft_consensus.cc:504] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
I20260809 05:24:57.221616 14452 leader_election.cc:288] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 election: Requested vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507)
I20260809 05:24:57.222126 14349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c89d43a474c5aa851f21f72672298" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bab1fa28d44acea582accfe6177f67"
I20260809 05:24:57.222213 13934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824c89d43a474c5aa851f21f72672298" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d9ef40a654fe283363b75c5db3898"
I20260809 05:24:57.222326 14349 raft_consensus.cc:3037] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:57.222438 13934 raft_consensus.cc:3037] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:24:57.224624 14349 raft_consensus.cc:2454] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 1.
I20260809 05:24:57.224706 13934 raft_consensus.cc:2454] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 1.
I20260809 05:24:57.225072 14040 leader_election.cc:302] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67; no voters:
I20260809 05:24:57.225417 14452 raft_consensus.cc:2781] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:24:57.225934 14452 raft_consensus.cc:686] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa57b5f27064426afb0bc9e11915869, State: Running, Role: LEADER
I20260809 05:24:57.226473 14452 consensus_queue.cc:237] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } }
W20260809 05:24:57.229298 14215 tablet.cc:2369] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:57.230731 13720 catalog_manager.cc:5495] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "0aa57b5f27064426afb0bc9e11915869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } health_report { overall_health: UNKNOWN } } }
W20260809 05:24:57.263252 14407 tablet.cc:2369] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:24:57.269402 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.264355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.22.1.66:46781)
0809 05:24:57.264582 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.264587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.264607 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:24:57.264702 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.264705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.264786 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.264908 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:24:57.265246 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.265963 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.267385 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.267790 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.267877 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.268351 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:24:57.268870 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:24:57.268979 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.269027 (+ 48us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.269094 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:24:57.277613 13934 raft_consensus.cc:1261] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Refusing update from remote peer 0aa57b5f27064426afb0bc9e11915869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:57.277653 14349 raft_consensus.cc:1261] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Refusing update from remote peer 0aa57b5f27064426afb0bc9e11915869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:24:57.278602 14459 consensus_queue.cc:1035] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:57.278951 14452 consensus_queue.cc:1035] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:24:57.306139 13719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:57.306494 13719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:24:57.311146 13719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Sending DeleteTablet for 3 replicas of tablet 824c89d43a474c5aa851f21f72672298
I20260809 05:24:57.312600 14324 tablet_service.cc:1514] Processing DeleteTablet for tablet 824c89d43a474c5aa851f21f72672298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:57 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:58874
I20260809 05:24:57.312589 13901 tablet_service.cc:1514] Processing DeleteTablet for tablet 824c89d43a474c5aa851f21f72672298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:57 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:37950
I20260809 05:24:57.312644 14106 tablet_service.cc:1514] Processing DeleteTablet for tablet 824c89d43a474c5aa851f21f72672298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:24:57 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:41640
I20260809 05:24:57.315259 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 13809
I20260809 05:24:57.328361 14488 tablet_replica.cc:330] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: stopping tablet replica
I20260809 05:24:57.329146 14488 raft_consensus.cc:2229] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:24:57.330078 14488 raft_consensus.cc:2258] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:57.331084 14485 tablet_replica.cc:330] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: stopping tablet replica
I20260809 05:24:57.331635 14485 raft_consensus.cc:2229] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:57.332237 14485 raft_consensus.cc:2258] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:57.332609 14488 ts_tablet_manager.cc:1905] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:57.332864 14486 tablet_replica.cc:330] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: stopping tablet replica
I20260809 05:24:57.333343 14486 raft_consensus.cc:2229] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:24:57.334215 14486 raft_consensus.cc:2258] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:24:57.336339 14485 ts_tablet_manager.cc:1905] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:57.341638 14488 ts_tablet_manager.cc:1918] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:57.342013 14488 log.cc:1196] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/824c89d43a474c5aa851f21f72672298
I20260809 05:24:57.342716 14488 ts_tablet_manager.cc:1939] T 824c89d43a474c5aa851f21f72672298 P 0aa57b5f27064426afb0bc9e11915869: Deleting consensus metadata
I20260809 05:24:57.342834 14486 ts_tablet_manager.cc:1905] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:24:57.344826 13703 catalog_manager.cc:4841] TS 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): tablet 824c89d43a474c5aa851f21f72672298 (table test-table [id=fbd819e673454b7991757fea1b7f8d9a]) successfully deleted
I20260809 05:24:57.350384 14486 ts_tablet_manager.cc:1918] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:57.350752 14486 log.cc:1196] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/824c89d43a474c5aa851f21f72672298
I20260809 05:24:57.351423 14486 ts_tablet_manager.cc:1939] T 824c89d43a474c5aa851f21f72672298 P 981d9ef40a654fe283363b75c5db3898: Deleting consensus metadata
I20260809 05:24:57.351423 14485 ts_tablet_manager.cc:1918] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:24:57.351687 14485 log.cc:1196] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/824c89d43a474c5aa851f21f72672298
Aug 09 05:24:57 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:24:57.352286 14485 ts_tablet_manager.cc:1939] T 824c89d43a474c5aa851f21f72672298 P a7bab1fa28d44acea582accfe6177f67: Deleting consensus metadata
I20260809 05:24:57.353639 13702 catalog_manager.cc:4841] TS 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507): tablet 824c89d43a474c5aa851f21f72672298 (table test-table [id=fbd819e673454b7991757fea1b7f8d9a]) successfully deleted
I20260809 05:24:57.354696 13703 catalog_manager.cc:4841] TS a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): tablet 824c89d43a474c5aa851f21f72672298 (table test-table [id=fbd819e673454b7991757fea1b7f8d9a]) successfully deleted
I20260809 05:24:57.359797 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.316542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.22.1.65:37507)
0809 05:24:57.316730 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.316737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.327380 (+ 10643us) server_negotiation.cc:407] Connection header received
0809 05:24:57.328600 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.328608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.328751 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.328845 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:57.339763 (+ 10918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.340922 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.343200 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.343874 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.343996 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.353311 (+ 9315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:57.353327 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:57.353349 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:57.353397 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:57.356577 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.357632 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.357639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.357642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.357787 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.358213 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.358217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.358219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.358639 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:57.358891 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.359300 (+ 409us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.359474 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:24:57.360602 13995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:57.450278 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 13809
I20260809 05:24:57.475553 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 14013
Aug 09 05:24:57 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:24:57.492074 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.476352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.22.1.66:46781)
0809 05:24:57.476527 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.476533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.476603 (+ 70us) server_negotiation.cc:407] Connection header received
0809 05:24:57.476804 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.476808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.476877 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.476984 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:57.478379 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.479233 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.480820 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.481317 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.481429 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.485833 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:57.485851 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:57.485866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:57.485916 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:57.488981 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.489753 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.489760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.489763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.489927 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.490314 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.490317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.490319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.490715 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:57.491061 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.491361 (+ 300us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.491595 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:24:57.492781 14204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:57.608417 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 14013
I20260809 05:24:57.633008 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 14225
Aug 09 05:24:57 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:24:57.649704 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:24:57.633984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.22.1.67:44011)
0809 05:24:57.634153 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0809 05:24:57.634157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:24:57.634231 (+ 74us) server_negotiation.cc:407] Connection header received
0809 05:24:57.634382 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:24:57.634386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:24:57.634453 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:24:57.634577 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:24:57.636061 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.636830 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.638273 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:24:57.638707 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:24:57.638808 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:24:57.643543 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:24:57.643562 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:24:57.643581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:24:57.643623 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:24:57.646562 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.647424 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.647431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.647434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.647627 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:24:57.648008 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:24:57.648011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:24:57.648013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:24:57.648471 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:24:57.648730 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0809 05:24:57.649113 (+ 383us) server_negotiation.cc:299] Negotiation successful
0809 05:24:57.649391 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:24:57.650427 14401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:57.738971 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 14225
I20260809 05:24:57.763060 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 13676
I20260809 05:24:57.764236 13768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:24:57.874080 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 13676
I20260809 05:25:17.901294 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--webserver_interface=127.22.1.126
--webserver_port=41011
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:25:18.241400 14496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:18.241724 14496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:18.241868 14496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:18.249066 14496 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 05:25:18.249181 14496 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 05:25:18.249238 14496 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 05:25:18.249303 14496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:25:18.249379 14496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:18.249476 14496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:18.249534 14496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:25:18.249606 14496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:25:18.259425 14496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=41011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:18.260732 14496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:18.262092 14496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:18.285670 14503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:18.285697 14505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:18.285915 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:18.286687 14496 server_base.cc:1034] running on GCE node
I20260809 05:25:18.287572 14496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:18.289278 14496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:18.290557 14496 hybrid_clock.cc:648] HybridClock initialized: now 1786253118290485 us; error 42 us; skew 500 ppm
I20260809 05:25:18.290983 14496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:18 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:18.299046 14496 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:25:18.300804 14496 webserver.cc:460] Webserver started at http://127.22.1.126:41011/ using document root <none> and password file <none>
I20260809 05:25:18.301690 14496 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:18.301822 14496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:18.308135 14496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260809 05:25:18.311604 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:18.313388 14496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:25:18.313639 14496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "dd54a710d4f844019f0974de7832bb7d"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "497822f90548f5e136029da5c1b2012f"
server_key_iv: "79a291c570dfef535e5dbf8a8b2d900b"
server_key_version: "encryptionkey@0"
I20260809 05:25:18.314764 14496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:18.330718 14496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:18.335330 14496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:18.335641 14496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:18.354187 14496 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38883
I20260809 05:25:18.354212 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38883 every 8 connection(s)
I20260809 05:25:18.355896 14496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:25:18.361223 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 14496
I20260809 05:25:18.362076 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:37507
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=37801
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:25:18.363191 14585 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:25:18.366565 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap starting.
I20260809 05:25:18.389321 14585 log.cc:824] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:18.402577 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:25:18.403136 14585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap complete.
I20260809 05:25:18.412215 14585 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:18.413090 14585 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Initialized, Role: FOLLOWER
I20260809 05:25:18.413750 14585 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:18.413987 14585 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:25:18.414106 14585 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:25:18.414274 14585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 1 FOLLOWER]: Advancing to term 2
I20260809 05:25:18.418237 14585 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:18.418844 14585 leader_election.cc:302] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd54a710d4f844019f0974de7832bb7d; no voters:
I20260809 05:25:18.419502 14585 leader_election.cc:288] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260809 05:25:18.419704 14589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Leader election won for term 2
I20260809 05:25:18.420882 14589 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 LEADER]: Becoming Leader. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Running, Role: LEADER
I20260809 05:25:18.421550 14589 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:18.421963 14585 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:25:18.424203 14593 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd54a710d4f844019f0974de7832bb7d. Latest consensus state: current_term: 2 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:25:18.424258 14590 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:25:18.424620 14593 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:25:18.424651 14590 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:25:18.425680 14601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:25:18.446537 14601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:25:18.448416 14601 catalog_manager.cc:1259] Loaded cluster ID: 5a0badd7cc624474843a21caf2f6b22a
I20260809 05:25:18.448618 14601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:25:18.453225 14601 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:25:18.455224 14601 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Loaded TSK: 0
I20260809 05:25:18.490530 14601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Generated new TSK 1
I20260809 05:25:18.490743 14616 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 05:25:18.490974 14601 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260809 05:25:18.746532 14587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:18.746851 14587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:18.746989 14587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:18.754724 14587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:18.754869 14587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:18.755021 14587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:25:18.765316 14587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:37507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=37801
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:18.766665 14587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:18.768046 14587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:18.783504 14626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:18.783686 14625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:18.783856 14631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:18.784763 14587 server_base.cc:1034] running on GCE node
I20260809 05:25:18.785497 14587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:18.786594 14587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:18.787848 14587 hybrid_clock.cc:648] HybridClock initialized: now 1786253118787780 us; error 49 us; skew 500 ppm
I20260809 05:25:18.788175 14587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:18 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:18.791484 14587 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:25:18.794375 14587 webserver.cc:460] Webserver started at http://127.22.1.65:37801/ using document root <none> and password file <none>
I20260809 05:25:18.795161 14587 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:18.795301 14587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:18.801173 14587 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260809 05:25:18.804070 14640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:18.805599 14587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:25:18.805751 14587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "981d9ef40a654fe283363b75c5db3898"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "4f634c021fc5321003e9c8f0f0f082fc"
server_key_iv: "7744a9639105dcfdb70743368fed0aa3"
server_key_version: "encryptionkey@0"
I20260809 05:25:18.806749 14587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:18.824389 14587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:18.828985 14587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:18.829303 14587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:18.830456 14587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:18.831982 14587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:25:18.832093 14587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:18.832204 14587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:25:18.832283 14587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:18.864187 14587 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:37507
I20260809 05:25:18.864243 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:37507 every 8 connection(s)
I20260809 05:25:18.865731 14587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:25:18.875020 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 14587
I20260809 05:25:18.875627 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:46781
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=37333
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:25:18 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253118, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:25:18.890563 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:18.868833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:43953)
0809 05:25:18.869693 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:18.869742 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:18.871584 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:18.873119 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:18.873127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:18.873762 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:25:18.874523 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:18.874546 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:18.876402 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:18.876406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:18.877748 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:18.877762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:18.878843 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:18.878850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:18.879138 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:18.879792 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:25:18.879816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:25:18.882207 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:25:18.886211 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:18.886221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:18.886239 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:18.886870 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:18.887325 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:18.887328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:18.887331 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:18.887554 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:18.888448 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:25:18.888456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:25:18.888785 (+ 329us) client_negotiation.cc:769] Sending connection context
0809 05:25:18.889028 (+ 243us) client_negotiation.cc:240] Negotiation successful
0809 05:25:18.889502 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
I20260809 05:25:18.890929 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:18.868668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:43953 (local address 127.22.1.126:38883)
0809 05:25:18.869344 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:18.869358 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:18.869883 (+ 525us) server_negotiation.cc:407] Connection header received
0809 05:25:18.872211 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:18.872225 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:18.872615 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:18.872941 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:18.874752 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:18.876219 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:18.877892 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:18.878603 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:18.878784 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:18.882531 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:18.882542 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:18.882561 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:18.882614 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:18.886055 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:18.887033 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:18.887041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:18.887048 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:18.887215 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:18.887673 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:18.887678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:18.887680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:18.888318 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:18.888500 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:18.889334 (+ 834us) server_negotiation.cc:299] Negotiation successful
0809 05:25:18.889785 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":276,"threads_started":1}
I20260809 05:25:18.893761 14821 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:25:18.894071 14821 heartbeater.cc:464] Registering TS with master...
I20260809 05:25:18.894816 14821 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:25:18.897084 14540 ts_manager.cc:194] Registered new tserver with Master: 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507)
I20260809 05:25:18.898754 14540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:43953
W20260809 05:25:19.225263 14826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:19.225577 14826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:19.225719 14826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:19.233799 14826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:19.233979 14826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:19.234144 14826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:25:19.244402 14826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:46781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=37333
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:19.245921 14826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:19.247524 14826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:19.261503 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:19.261561 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:19.261445 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:19.262750 14826 server_base.cc:1034] running on GCE node
I20260809 05:25:19.263392 14826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:19.264389 14826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:19.265599 14826 hybrid_clock.cc:648] HybridClock initialized: now 1786253119265552 us; error 39 us; skew 500 ppm
I20260809 05:25:19.265928 14826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:19 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:19.269176 14826 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:25:19.270867 14826 webserver.cc:460] Webserver started at http://127.22.1.66:37333/ using document root <none> and password file <none>
I20260809 05:25:19.271638 14826 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:19.271785 14826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:19.277697 14826 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:25:19.280550 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.282061 14826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:25:19.282226 14826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "0aa57b5f27064426afb0bc9e11915869"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "4a1f19d4ac3c425c51cda8e153f7b08b"
server_key_iv: "70586e092101030c29d918c3d4246f63"
server_key_version: "encryptionkey@0"
I20260809 05:25:19.283215 14826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:19.306191 14826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:19.310643 14826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:19.310931 14826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:19.312088 14826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:19.313624 14826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:25:19.313720 14826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.313838 14826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:25:19.313900 14826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.345777 14826 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:46781
I20260809 05:25:19.345860 15022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:46781 every 8 connection(s)
I20260809 05:25:19.347316 14826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:25:19.357453 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 14826
I20260809 05:25:19.358235 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:44011
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=41555
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 09 05:25:19 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:25:19.371717 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.350072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:55533 (local address 127.22.1.126:38883)
0809 05:25:19.350283 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.350289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.351721 (+ 1432us) server_negotiation.cc:407] Connection header received
0809 05:25:19.353837 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.353844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.353962 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.354090 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:19.355739 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.356864 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.359292 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.359988 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.360114 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.364717 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:19.364725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:19.364730 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:19.364774 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:19.368280 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.369253 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.369258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.369260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.369399 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.369877 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.369881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.369883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.370195 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:19.370332 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.371096 (+ 764us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.371338 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:25:19.372831 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.350702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:55533)
0809 05:25:19.351527 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.351578 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.353449 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.354328 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.354337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.354923 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:25:19.355550 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.355569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.357092 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.357099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.359135 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.359152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.360256 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.360263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.360568 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.361422 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:25:19.361468 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:25:19.364394 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:25:19.368454 (+ 4060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:19.368463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:19.368483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:19.369076 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:19.369543 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:19.369546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:19.369549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:19.369739 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:19.370332 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:25:19.370341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:25:19.370710 (+ 369us) client_negotiation.cc:769] Sending connection context
0809 05:25:19.370967 (+ 257us) client_negotiation.cc:240] Negotiation successful
0809 05:25:19.371558 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":352,"threads_started":1}
I20260809 05:25:19.374377 15023 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:25:19.374686 15023 heartbeater.cc:464] Registering TS with master...
I20260809 05:25:19.375486 15023 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:25:19.377110 14540 ts_manager.cc:194] Registered new tserver with Master: 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
I20260809 05:25:19.377955 14540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:55533
W20260809 05:25:19.706379 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:19.706696 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:19.706831 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:19.714331 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:19.714504 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:19.714675 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:25:19.725168 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:44011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=41555
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:19.726575 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:19.728014 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:19.741269 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:19.741340 15038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:19.741313 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:19.742112 15030 server_base.cc:1034] running on GCE node
I20260809 05:25:19.742691 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:19.743624 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:19.744822 15030 hybrid_clock.cc:648] HybridClock initialized: now 1786253119744769 us; error 45 us; skew 500 ppm
I20260809 05:25:19.745126 15030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:19 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:19.748117 15030 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:25:19.749668 15030 webserver.cc:460] Webserver started at http://127.22.1.67:41555/ using document root <none> and password file <none>
I20260809 05:25:19.750424 15030 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:19.750554 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:19.756758 15030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260809 05:25:19.759799 15048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.761318 15030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:25:19.761482 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "a7bab1fa28d44acea582accfe6177f67"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "23797552404c17cc5caae0ac0078d52e"
server_key_iv: "925f99979d90d762939aef225de132bd"
server_key_version: "encryptionkey@0"
I20260809 05:25:19.762475 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:19.799302 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:19.803553 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:19.803822 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:19.804872 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:19.806252 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:25:19.806350 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.806457 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:25:19.806535 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:19.837721 15030 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:44011
I20260809 05:25:19.837810 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:44011 every 8 connection(s)
I20260809 05:25:19.839243 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:25:19.840593 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 15030
Aug 09 05:25:19 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253119, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:25:19.856878 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.841836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.22.1.126:38883)
0809 05:25:19.842079 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.842085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.842485 (+ 400us) server_negotiation.cc:407] Connection header received
0809 05:25:19.842713 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.842724 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.842815 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.842929 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:19.844548 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.845368 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.847182 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.847840 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.847991 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.850978 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:19.850985 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:19.850990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:19.851034 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:19.853970 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.854686 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.854691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.854693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.854819 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.855279 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.855286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.855289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.855679 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:19.855940 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.856241 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.856472 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 05:25:19.862252 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.842564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:59335 (local address 127.22.1.126:38883)
0809 05:25:19.843021 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.843026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.844381 (+ 1355us) server_negotiation.cc:407] Connection header received
0809 05:25:19.846556 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.846561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.846651 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.846736 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:19.849077 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.849921 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.851849 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.852341 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.852465 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.856491 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:19.856497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:19.856502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:19.856537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:19.859392 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.860175 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.860179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.860181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.860301 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.860680 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.860683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.860685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.860955 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:19.861062 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.861791 (+ 729us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.861966 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260809 05:25:19.863258 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.843351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:59335)
0809 05:25:19.844202 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.844252 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.846186 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.847326 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.847339 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.848144 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:25:19.848875 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.848897 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.850076 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.850080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.851711 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.851725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.852590 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.852597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.852849 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.853535 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:25:19.853571 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:25:19.856180 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:25:19.859557 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:19.859568 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:19.859588 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:19.860067 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:19.860381 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:19.860384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:19.860386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:19.860599 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:19.861048 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:25:19.861054 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:25:19.861434 (+ 380us) client_negotiation.cc:769] Sending connection context
0809 05:25:19.861702 (+ 268us) client_negotiation.cc:240] Negotiation successful
0809 05:25:19.862115 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":340,"threads_started":1}
I20260809 05:25:19.867377 15214 heartbeater.cc:347] Connected to a master server at 127.22.1.126:38883
I20260809 05:25:19.867715 15214 heartbeater.cc:464] Registering TS with master...
I20260809 05:25:19.868489 15214 heartbeater.cc:511] Master 127.22.1.126:38883 requested a full tablet report, sending...
I20260809 05:25:19.869983 14540 ts_manager.cc:194] Registered new tserver with Master: a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011)
I20260809 05:25:19.870858 14540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:59335
I20260809 05:25:19.872150 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:25:19.879073 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.873207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.22.1.126:38883)
0809 05:25:19.873359 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.873364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.874031 (+ 667us) server_negotiation.cc:407] Connection header received
0809 05:25:19.874206 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.874210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.874364 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.874483 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:25:19.874879 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.875616 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.877266 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.877671 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.877763 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.878229 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:25:19.878548 (+ 319us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0809 05:25:19.878863 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56770: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260809 05:25:19.879194 15219 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56770: token expired
W20260809 05:25:19.879730 14422 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.22.1.126:38883)
I20260809 05:25:19.892246 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.880197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.22.1.126:38883)
0809 05:25:19.880334 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.880338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.880406 (+ 68us) server_negotiation.cc:407] Connection header received
0809 05:25:19.880530 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.880534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.880588 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.880662 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:19.882168 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.882896 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.884350 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.884745 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.884830 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.887600 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:19.887605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:19.887609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:19.887642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:19.890029 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.890676 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.890679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.890682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.890795 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:19.891105 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:19.891108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:19.891109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:19.891406 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:19.891621 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.891846 (+ 225us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.892006 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 05:25:19.895783 14540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260809 05:25:19.902200 14821 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
W20260809 05:25:19.903481 14540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:25:19.940490 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.931647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.22.1.66:46781)
0809 05:25:19.932287 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.932295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.932316 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:25:19.932416 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.932423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.932657 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.932855 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:25:19.933867 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.934845 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.937132 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.938270 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.938402 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.938609 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.940024 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.940194 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":363,"threads_started":1}
I20260809 05:25:19.940485 15227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.916234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43420)
0809 05:25:19.929826 (+ 13592us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.929881 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.930114 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.931984 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.931998 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.932033 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:25:19.932482 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.932497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.935113 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.935119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.937734 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.937750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.939430 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.939437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.939596 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.939662 (+ 66us) client_negotiation.cc:769] Sending connection context
0809 05:25:19.939813 (+ 151us) client_negotiation.cc:240] Negotiation successful
0809 05:25:19.940001 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13257,"mutex_wait_us":73,"thread_start_us":13054,"threads_started":1}
I20260809 05:25:19.940531 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.931403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:46781 (local address 127.0.0.1:60534)
0809 05:25:19.932008 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.932036 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.932161 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.933104 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.933108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.933132 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:25:19.933736 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.933747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.934984 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.934990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.936983 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.936995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.938884 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.938896 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.939834 (+ 938us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.939869 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 05:25:19.939950 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:25:19.940078 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":52,"thread_start_us":325,"threads_started":1}
I20260809 05:25:19.940587 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.916985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.22.1.67:44011)
0809 05:25:19.930811 (+ 13826us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.930821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.931551 (+ 730us) server_negotiation.cc:407] Connection header received
0809 05:25:19.932453 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.932460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.932657 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.932856 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:25:19.933841 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.935410 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.938147 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.939467 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.939588 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.939809 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.940111 (+ 302us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.940276 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13654,"thread_start_us":13510,"threads_started":1}
I20260809 05:25:19.941283 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.916305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44011 (local address 127.0.0.1:46734)
0809 05:25:19.931273 (+ 14968us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.931296 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.931413 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.933120 (+ 1707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.933124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.933141 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:25:19.933641 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.933657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.935602 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.935607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.938012 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.938023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.939750 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.939756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:19.939908 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.939938 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 05:25:19.940771 (+ 833us) client_negotiation.cc:240] Negotiation successful
0809 05:25:19.940902 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14847,"mutex_wait_us":30,"spinlock_wait_cycles":3456,"thread_start_us":14695,"threads_started":1}
I20260809 05:25:19.941464 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.916351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43420 (local address 127.22.1.65:37507)
0809 05:25:19.931247 (+ 14896us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.931257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.931290 (+ 33us) server_negotiation.cc:407] Connection header received
0809 05:25:19.931400 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.931408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.931709 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.932862 (+ 1153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:25:19.933125 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.934924 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.937906 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.939222 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.939346 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:19.939573 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:19.940957 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0809 05:25:19.941137 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14725,"thread_start_us":14567,"threads_started":1}
I20260809 05:25:19.944557 14728 tablet_service.cc:1467] Processing CreateTablet for tablet c5b486373730421193aae243eab996af (DEFAULT_TABLE table=test-table [id=da4f0d3615e1404bb8439f3807849e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:25:19.944525 15120 tablet_service.cc:1467] Processing CreateTablet for tablet c5b486373730421193aae243eab996af (DEFAULT_TABLE table=test-table [id=da4f0d3615e1404bb8439f3807849e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:25:19.945829 15120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b486373730421193aae243eab996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:25:19.945932 14728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b486373730421193aae243eab996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:25:19.947778 14920 tablet_service.cc:1467] Processing CreateTablet for tablet c5b486373730421193aae243eab996af (DEFAULT_TABLE table=test-table [id=da4f0d3615e1404bb8439f3807849e57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:25:19.949115 14920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b486373730421193aae243eab996af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:25:19.958627 15237 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Bootstrap starting.
I20260809 05:25:19.961692 15238 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Bootstrap starting.
I20260809 05:25:19.962386 15237 tablet_bootstrap.cc:654] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Neither blocks nor log segments found. Creating new log.
I20260809 05:25:19.963745 15237 log.cc:824] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:19.964121 15239 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Bootstrap starting.
I20260809 05:25:19.965354 15238 tablet_bootstrap.cc:654] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Neither blocks nor log segments found. Creating new log.
I20260809 05:25:19.966676 15238 log.cc:824] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:19.967669 15239 tablet_bootstrap.cc:654] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Neither blocks nor log segments found. Creating new log.
I20260809 05:25:19.967931 15237 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: No bootstrap required, opened a new log
I20260809 05:25:19.968230 15237 ts_tablet_manager.cc:1397] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 05:25:19.968865 15239 log.cc:824] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:19.969482 15238 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: No bootstrap required, opened a new log
I20260809 05:25:19.969763 15238 ts_tablet_manager.cc:1397] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260809 05:25:19.971911 15239 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: No bootstrap required, opened a new log
I20260809 05:25:19.972170 15239 ts_tablet_manager.cc:1397] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 05:25:19.976785 15238 raft_consensus.cc:348] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.977205 15238 raft_consensus.cc:374] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:25:19.977303 15238 raft_consensus.cc:729] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981d9ef40a654fe283363b75c5db3898, State: Initialized, Role: FOLLOWER
I20260809 05:25:19.977892 15238 consensus_queue.cc:260] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.978281 15237 raft_consensus.cc:348] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.978716 15237 raft_consensus.cc:374] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:25:19.978806 15237 raft_consensus.cc:729] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7bab1fa28d44acea582accfe6177f67, State: Initialized, Role: FOLLOWER
I20260809 05:25:19.979390 15237 consensus_queue.cc:260] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.980129 15238 ts_tablet_manager.cc:1428] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260809 05:25:19.981532 15214 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
I20260809 05:25:19.982232 15237 ts_tablet_manager.cc:1428] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260809 05:25:19.983064 15239 raft_consensus.cc:348] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.983573 15239 raft_consensus.cc:374] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:25:19.983685 15239 raft_consensus.cc:729] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aa57b5f27064426afb0bc9e11915869, State: Initialized, Role: FOLLOWER
I20260809 05:25:19.984318 15239 consensus_queue.cc:260] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.986222 15023 heartbeater.cc:503] Master 127.22.1.126:38883 was elected leader, sending a full tablet report...
I20260809 05:25:19.986886 15239 ts_tablet_manager.cc:1428] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260809 05:25:19.993854 15246 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:19.994151 15246 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:19.995972 15246 leader_election.cc:288] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507), a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011)
I20260809 05:25:20.004837 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.996747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:37015 (local address 127.22.1.67:44011)
0809 05:25:19.997005 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.997012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.997640 (+ 628us) server_negotiation.cc:407] Connection header received
0809 05:25:19.997853 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.997860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.997938 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.998008 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:25:19.998661 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.999726 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.002380 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.003596 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.003728 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.003840 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.004424 (+ 584us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.004575 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:25:20.004834 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.996468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:37725 (local address 127.22.1.65:37507)
0809 05:25:19.996686 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:19.996693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:19.997765 (+ 1072us) server_negotiation.cc:407] Connection header received
0809 05:25:19.997911 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:19.997914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:19.997974 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:19.998077 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:25:19.998697 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.999615 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.002386 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.003582 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.003729 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.003841 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.004425 (+ 584us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.004575 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:25:20.005057 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.996241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44011 (local address 127.22.1.66:37015)
0809 05:25:19.997536 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.997552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.997691 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.998043 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.998048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.998073 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:25:19.998489 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.998502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.999891 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.999895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:20.002211 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:20.002222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.003922 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.003930 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:20.004205 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.004285 (+ 80us) client_negotiation.cc:769] Sending connection context
0809 05:25:20.004379 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 05:25:20.004589 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"mutex_wait_us":312,"thread_start_us":958,"threads_started":1}
I20260809 05:25:20.005216 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:19.996297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.22.1.66:37725)
0809 05:25:19.997665 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:19.997679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:19.997798 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:19.998106 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:19.998110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:19.998140 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:25:19.998535 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:19.998545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:19.999774 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:19.999780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:20.002203 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:20.002222 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.003922 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.003936 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:20.004205 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.004284 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 05:25:20.004380 (+ 96us) client_negotiation.cc:240] Negotiation successful
0809 05:25:20.004588 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1258,"mutex_wait_us":133,"thread_start_us":965,"threads_started":1}
I20260809 05:25:20.006214 14758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b486373730421193aae243eab996af" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d9ef40a654fe283363b75c5db3898" is_pre_election: true
I20260809 05:25:20.006214 15150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b486373730421193aae243eab996af" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bab1fa28d44acea582accfe6177f67" is_pre_election: true
I20260809 05:25:20.006805 15150 raft_consensus.cc:2454] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 0.
I20260809 05:25:20.006805 14758 raft_consensus.cc:2454] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 0.
I20260809 05:25:20.007570 14857 leader_election.cc:302] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa57b5f27064426afb0bc9e11915869, 981d9ef40a654fe283363b75c5db3898; no voters:
I20260809 05:25:20.007989 15246 raft_consensus.cc:2781] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:25:20.008167 15246 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:25:20.008268 15246 raft_consensus.cc:3037] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:25:20.010524 15246 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:20.011188 15246 leader_election.cc:288] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 election: Requested vote from peers 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507), a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011)
I20260809 05:25:20.011644 14758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b486373730421193aae243eab996af" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "981d9ef40a654fe283363b75c5db3898"
I20260809 05:25:20.011711 15150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b486373730421193aae243eab996af" candidate_uuid: "0aa57b5f27064426afb0bc9e11915869" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7bab1fa28d44acea582accfe6177f67"
I20260809 05:25:20.011830 14758 raft_consensus.cc:3037] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:25:20.011898 15150 raft_consensus.cc:3037] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:25:20.014042 14758 raft_consensus.cc:2454] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 1.
I20260809 05:25:20.014138 15150 raft_consensus.cc:2454] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0aa57b5f27064426afb0bc9e11915869 in term 1.
I20260809 05:25:20.014405 14857 leader_election.cc:302] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aa57b5f27064426afb0bc9e11915869, 981d9ef40a654fe283363b75c5db3898; no voters:
I20260809 05:25:20.014670 15246 raft_consensus.cc:2781] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:25:20.015081 15246 raft_consensus.cc:686] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 1 LEADER]: Becoming Leader. State: Replica: 0aa57b5f27064426afb0bc9e11915869, State: Running, Role: LEADER
I20260809 05:25:20.015681 15246 consensus_queue.cc:237] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:20.020627 14539 catalog_manager.cc:5495] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "0aa57b5f27064426afb0bc9e11915869" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } health_report { overall_health: HEALTHY } } }
I20260809 05:25:20.052479 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.047227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.22.1.66:46781)
0809 05:25:20.047417 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:20.047422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:20.047440 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:25:20.047618 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:20.047622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:20.047701 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:20.047826 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:25:20.048135 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.048918 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.050426 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.050850 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.050946 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.051447 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:25:20.051968 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:25:20.052102 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.052140 (+ 38us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.052209 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:25:20.061627 14758 raft_consensus.cc:1261] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Refusing update from remote peer 0aa57b5f27064426afb0bc9e11915869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:25:20.061627 15150 raft_consensus.cc:1261] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Refusing update from remote peer 0aa57b5f27064426afb0bc9e11915869: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:25:20.062798 15252 consensus_queue.cc:1035] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:25:20.063148 15246 consensus_queue.cc:1035] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [LEADER]: Connected to new peer: Peer: permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:25:20.076784 15254 mvcc.cc:205] Tried to move back new op lower bound from 7316492779760324608 to 7316492779588800512. Current Snapshot: MvccSnapshot[applied={T|T < 7316492779760324608}]
I20260809 05:25:20.078841 15256 mvcc.cc:205] Tried to move back new op lower bound from 7316492779760324608 to 7316492779588800512. Current Snapshot: MvccSnapshot[applied={T|T < 7316492779760324608}]
I20260809 05:25:20.096845 14539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:25:20.097122 14539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:25:20.100539 14539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Sending DeleteTablet for 3 replicas of tablet c5b486373730421193aae243eab996af
I20260809 05:25:20.101711 15120 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b486373730421193aae243eab996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:25:20 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:46734
I20260809 05:25:20.101883 14728 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b486373730421193aae243eab996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:25:20 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:43420
I20260809 05:25:20.101980 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 14587
I20260809 05:25:20.102967 14920 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b486373730421193aae243eab996af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:25:20 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:60534
I20260809 05:25:20.123593 15229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.103691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.22.1.65:37507)
0809 05:25:20.103862 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:20.103868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:20.103886 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:25:20.103942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:20.103946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:20.104027 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:20.104128 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:20.106145 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.107388 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.109471 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.110219 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.110347 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.114582 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:20.114605 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:20.114631 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:20.114694 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:20.119087 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.120225 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.120235 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.120239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.120414 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.120918 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.120923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.120927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.121489 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:20.121890 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.122909 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.123159 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:25:20.125078 14809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:20.125577 15269 tablet_replica.cc:330] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: stopping tablet replica
I20260809 05:25:20.126180 15269 raft_consensus.cc:2229] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:25:20.126286 15267 tablet_replica.cc:330] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: stopping tablet replica
I20260809 05:25:20.126778 15267 raft_consensus.cc:2229] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:25:20.127264 15269 raft_consensus.cc:2258] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:25:20.127362 15267 raft_consensus.cc:2258] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:25:20.129834 15269 ts_tablet_manager.cc:1905] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:25:20.129894 15267 ts_tablet_manager.cc:1905] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:25:20.137728 15269 ts_tablet_manager.cc:1918] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:25:20.138095 15269 log.cc:1196] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/c5b486373730421193aae243eab996af
I20260809 05:25:20.138607 15267 ts_tablet_manager.cc:1918] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:25:20.138782 15267 log.cc:1196] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/c5b486373730421193aae243eab996af
I20260809 05:25:20.139081 15269 ts_tablet_manager.cc:1939] T c5b486373730421193aae243eab996af P 0aa57b5f27064426afb0bc9e11915869: Deleting consensus metadata
I20260809 05:25:20.139328 15267 ts_tablet_manager.cc:1939] T c5b486373730421193aae243eab996af P a7bab1fa28d44acea582accfe6177f67: Deleting consensus metadata
I20260809 05:25:20.141361 14522 catalog_manager.cc:4841] TS a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): tablet c5b486373730421193aae243eab996af (table test-table [id=da4f0d3615e1404bb8439f3807849e57]) successfully deleted
I20260809 05:25:20.141767 14522 catalog_manager.cc:4841] TS 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): tablet c5b486373730421193aae243eab996af (table test-table [id=da4f0d3615e1404bb8439f3807849e57]) successfully deleted
I20260809 05:25:20.240079 15268 tablet_replica.cc:330] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: stopping tablet replica
I20260809 05:25:20.240609 15268 raft_consensus.cc:2229] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:25:20.241010 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 14587
I20260809 05:25:20.241180 15268 raft_consensus.cc:2258] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260809 05:25:20.266825 14521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:37507 (error 108)
I20260809 05:25:20.267560 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 14826
I20260809 05:25:20.268885 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.268360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43442)
0809 05:25:20.268618 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.268722 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260809 05:25:20.269140 14521 catalog_manager.cc:4584] TS 981d9ef40a654fe283363b75c5db3898 (127.22.1.65:37507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5b486373730421193aae243eab996af: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
I20260809 05:25:20.288802 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.268371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.22.1.66:46781)
0809 05:25:20.268554 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:20.268560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:20.268754 (+ 194us) server_negotiation.cc:407] Connection header received
0809 05:25:20.268936 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:20.268943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:20.269060 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:20.269190 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:20.270946 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.271762 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.273829 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.274312 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.274414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.282715 (+ 8301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:20.282729 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:20.282747 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:20.282790 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:20.285848 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.286743 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.286750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.286753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.286945 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.287318 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.287322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.287324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.287777 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:20.288070 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.288350 (+ 280us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.288517 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:25:20.289546 15014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:20.323105 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.322485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43446)
0809 05:25:20.322718 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.322854 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 05:25:20.324314 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.323898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43460)
0809 05:25:20.324069 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.324177 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260809 05:25:20.384883 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.384326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43476)
0809 05:25:20.384571 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.384686 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 05:25:20.386044 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.385629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43488)
0809 05:25:20.385784 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.385875 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 05:25:20.405316 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 14826
I20260809 05:25:20.433446 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 15030
I20260809 05:25:20.447158 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.434258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.22.1.67:44011)
0809 05:25:20.434423 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:20.434427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:20.434446 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:25:20.434601 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:20.434605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:20.434669 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:20.434777 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:20.436125 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.436879 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.438300 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:20.438733 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:20.438836 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:20.441555 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:20.441571 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:20.441591 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:20.441627 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:20.444321 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.445240 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.445248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.445251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.445398 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:20.445753 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:20.445756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:20.445759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:20.446151 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:20.446381 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:20.446643 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 05:25:20.446831 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:25:20.447882 15204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:20.466845 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.466248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43492)
0809 05:25:20.466517 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.466656 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 05:25:20.468060 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.467583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43506)
0809 05:25:20.467823 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.467913 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260809 05:25:20.537624 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 15030
I20260809 05:25:20.560510 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 14496
I20260809 05:25:20.561715 14578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:20.675844 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:20.675109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:37507 (local address 127.0.0.1:43512)
0809 05:25:20.675450 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:20.675544 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260809 05:25:20.676074 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 14496
I20260809 05:25:40.703402 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--webserver_interface=127.22.1.126
--webserver_port=41011
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:25:41.049377 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:41.049711 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:41.049852 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:41.057320 15276 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260809 05:25:41.057436 15276 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260809 05:25:41.057472 15276 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260809 05:25:41.057575 15276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:25:41.057651 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:41.057746 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:41.057803 15276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:25:41.057878 15276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:25:41.069706 15276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=41011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:41.071164 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:41.072595 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:41.085449 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:41.085580 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:41.085523 15292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:41.086253 15276 server_base.cc:1034] running on GCE node
I20260809 05:25:41.087178 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:41.088797 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:41.090052 15276 hybrid_clock.cc:648] HybridClock initialized: now 1786253141089984 us; error 48 us; skew 500 ppm
I20260809 05:25:41.090402 15276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:41 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253141, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:41.097702 15276 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:25:41.099437 15276 webserver.cc:460] Webserver started at http://127.22.1.126:41011/ using document root <none> and password file <none>
I20260809 05:25:41.100216 15276 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:41.100335 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:41.106256 15276 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260809 05:25:41.109261 15302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:41.110627 15276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 05:25:41.110803 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "dd54a710d4f844019f0974de7832bb7d"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "497822f90548f5e136029da5c1b2012f"
server_key_iv: "79a291c570dfef535e5dbf8a8b2d900b"
server_key_version: "encryptionkey@0"
I20260809 05:25:41.111786 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:41.138959 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:41.143534 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:41.143822 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:41.162045 15276 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38883
I20260809 05:25:41.162096 15383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38883 every 8 connection(s)
I20260809 05:25:41.163717 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:25:41.171304 15384 sys_catalog.cc:263] Verifying existing consensus state
I20260809 05:25:41.173924 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 15276
I20260809 05:25:41.175025 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:37507
--local_ip_for_outbound_sockets=127.22.1.65
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=37801
--webserver_interface=127.22.1.65
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:25:41.175525 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap starting.
I20260809 05:25:41.198753 15384 log.cc:824] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:41.216058 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:25:41.216665 15384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Bootstrap complete.
I20260809 05:25:41.226275 15384 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:41.227197 15384 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Initialized, Role: FOLLOWER
I20260809 05:25:41.227815 15384 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:41.228029 15384 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:25:41.228144 15384 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:25:41.228302 15384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 2 FOLLOWER]: Advancing to term 3
I20260809 05:25:41.232148 15384 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:41.232671 15384 leader_election.cc:302] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd54a710d4f844019f0974de7832bb7d; no voters:
I20260809 05:25:41.233359 15384 leader_election.cc:288] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260809 05:25:41.233518 15389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 3 FOLLOWER]: Leader election won for term 3
I20260809 05:25:41.234763 15389 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [term 3 LEADER]: Becoming Leader. State: Replica: dd54a710d4f844019f0974de7832bb7d, State: Running, Role: LEADER
I20260809 05:25:41.235378 15389 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } }
I20260809 05:25:41.235765 15384 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:25:41.238014 15390 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:25:41.238415 15390 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:25:41.238888 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd54a710d4f844019f0974de7832bb7d. Latest consensus state: current_term: 3 leader_uuid: "dd54a710d4f844019f0974de7832bb7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd54a710d4f844019f0974de7832bb7d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38883 } } }
I20260809 05:25:41.239192 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d [sys.catalog]: This master's current role is: LEADER
I20260809 05:25:41.250590 15398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:25:41.259613 15398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:25:41.261236 15398 catalog_manager.cc:1259] Loaded cluster ID: 5a0badd7cc624474843a21caf2f6b22a
I20260809 05:25:41.261600 15398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:25:41.265828 15398 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:25:41.267606 15398 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Loaded TSK: 0
I20260809 05:25:41.267884 15398 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dd54a710d4f844019f0974de7832bb7d: Loaded TSK: 1
F20260809 05:25:41.269285 15398 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9a2217f9ad google::LogMessage::Fail() at ??:0
@ 0x7f9a2218288a google::LogMessage::SendToLog() at ??:0
@ 0x7f9a2217f4de google::LogMessage::Flush() at ??:0
@ 0x7f9a22180a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a25ee37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9a25ed273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9a37bae1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9a37bad783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9a37c392a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9a37c390c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9a37c835e9 std::function<>::operator()() at ??:0
@ 0x7f9a37bb1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9a37bafc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a37c37610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9a37c37439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9a24dcb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a24dce91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9a24dce799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9a24da9eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a280836db start_thread at ??:0
@ 0x7f9a2101961f clone at ??:0
W20260809 05:25:41.568966 15386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:41.569360 15386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:41.569525 15386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:41.577064 15386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:41.577345 15386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:41.577543 15386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:25:41.587868 15386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:37507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=37801
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:41.589308 15386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:41.590655 15386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:41.603951 15448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:41.604031 15446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:41.604530 15445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:41.605005 15386 server_base.cc:1034] running on GCE node
I20260809 05:25:41.605731 15386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:41.606781 15386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:41.607982 15386 hybrid_clock.cc:648] HybridClock initialized: now 1786253141607914 us; error 65 us; skew 500 ppm
I20260809 05:25:41.608312 15386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:41 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253141, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:41.611550 15386 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:25:41.613088 15386 webserver.cc:460] Webserver started at http://127.22.1.65:37801/ using document root <none> and password file <none>
I20260809 05:25:41.613838 15386 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:41.613973 15386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:41.619648 15386 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260809 05:25:41.622712 15457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:41.624162 15386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:25:41.624339 15386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "981d9ef40a654fe283363b75c5db3898"
format_stamp: "Formatted at 2026-08-09 05:24:55 on dist-test-slave-rmhg"
server_key: "4f634c021fc5321003e9c8f0f0f082fc"
server_key_iv: "7744a9639105dcfdb70743368fed0aa3"
server_key_version: "encryptionkey@0"
I20260809 05:25:41.625413 15386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:41.659169 15386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:41.663797 15386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:41.664074 15386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:41.665220 15386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:41.667222 15469 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260809 05:25:41.673353 15386 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260809 05:25:41.673498 15386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260809 05:25:41.673655 15386 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260809 05:25:41.677001 15386 ts_tablet_manager.cc:610] Registered 1 tablets
I20260809 05:25:41.677093 15386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260809 05:25:41.677531 15469 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Bootstrap starting.
I20260809 05:25:41.705889 15469 log.cc:824] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Log is configured to *not* fsync() on all Append() calls
I20260809 05:25:41.716682 15386 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:37507
I20260809 05:25:41.716701 15619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:37507 every 8 connection(s)
I20260809 05:25:41.718075 15386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:25:41.719308 15469 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260809 05:25:41.719965 15469 tablet_bootstrap.cc:492] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Bootstrap complete.
I20260809 05:25:41.721848 15469 ts_tablet_manager.cc:1397] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Time spent bootstrapping tablet: real 0.045s user 0.029s sys 0.014s
I20260809 05:25:41.723255 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 15386
I20260809 05:25:41.724076 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:41.721188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:45465)
0809 05:25:41.722239 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:41.722538 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":507,"threads_started":1}
I20260809 05:25:41.724157 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:46781
--local_ip_for_outbound_sockets=127.22.1.66
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=37333
--webserver_interface=127.22.1.66
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:25:41.725090 15620 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:38883 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:38883: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
I20260809 05:25:41.726307 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:41.725818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:36351)
0809 05:25:41.726027 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:41.726143 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260809 05:25:41.727430 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:41.726947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:59941)
0809 05:25:41.727121 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:41.727239 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260809 05:25:41.727698 15620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 05:25:41.730495 15469 raft_consensus.cc:348] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:41.731392 15469 raft_consensus.cc:729] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981d9ef40a654fe283363b75c5db3898, State: Initialized, Role: FOLLOWER
I20260809 05:25:41.732008 15469 consensus_queue.cc:260] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:41.733760 15469 ts_tablet_manager.cc:1428] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260809 05:25:41.733970 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:41.733492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:52299)
0809 05:25:41.733645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:41.733731 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 05:25:41.734957 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:41.734597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:34883)
0809 05:25:41.734737 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:41.734816 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260809 05:25:42.066895 15624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:42.067236 15624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:42.067344 15624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:42.075461 15624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:42.075579 15624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:42.075700 15624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:25:42.085697 15624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:46781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=37333
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:42.087030 15624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:42.088370 15624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:42.101461 15630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:42.101500 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:42.101429 15631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:42.102131 15624 server_base.cc:1034] running on GCE node
I20260809 05:25:42.102746 15624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:42.103766 15624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:42.104956 15624 hybrid_clock.cc:648] HybridClock initialized: now 1786253142104901 us; error 50 us; skew 500 ppm
I20260809 05:25:42.105309 15624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:42 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253142, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:42.108403 15624 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:25:42.109922 15624 webserver.cc:460] Webserver started at http://127.22.1.66:37333/ using document root <none> and password file <none>
I20260809 05:25:42.110601 15624 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:42.110723 15624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:42.116209 15624 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260809 05:25:42.119035 15644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.120496 15624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:25:42.120680 15624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "0aa57b5f27064426afb0bc9e11915869"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "4a1f19d4ac3c425c51cda8e153f7b08b"
server_key_iv: "70586e092101030c29d918c3d4246f63"
server_key_version: "encryptionkey@0"
I20260809 05:25:42.121794 15624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:42.156392 15624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:42.160665 15624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:42.160912 15624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:42.161981 15624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:42.163475 15624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:25:42.163574 15624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.163666 15624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:25:42.163723 15624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.196765 15624 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:46781
I20260809 05:25:42.196846 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:46781 every 8 connection(s)
I20260809 05:25:42.198329 15624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:25:42.203732 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.201575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:56043)
0809 05:25:42.202390 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.202656 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":340,"threads_started":1}
W20260809 05:25:42.204664 15809 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:38883 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:38883: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
I20260809 05:25:42.205651 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 15624
I20260809 05:25:42.205709 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.205324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:58873)
0809 05:25:42.205508 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.205588 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260809 05:25:42.206281 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:44011
--local_ip_for_outbound_sockets=127.22.1.67
--tserver_master_addrs=127.22.1.126:38883
--webserver_port=41555
--webserver_interface=127.22.1.67
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:42331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:25:42.206657 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.206280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:57815)
0809 05:25:42.206422 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.206524 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260809 05:25:42.206894 15809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260809 05:25:42.566570 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:42.566887 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:42.567027 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:42.574571 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:42.574699 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:42.574858 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:25:42.584978 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:42331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:44011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=41555
--tserver_master_addrs=127.22.1.126:38883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:42.586333 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:42.587687 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:42.600728 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:42.600800 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:42.600828 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:42.601965 15816 server_base.cc:1034] running on GCE node
I20260809 05:25:42.602661 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:42.603701 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:42.604928 15816 hybrid_clock.cc:648] HybridClock initialized: now 1786253142604878 us; error 38 us; skew 500 ppm
I20260809 05:25:42.605350 15816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:42 dist-test-slave-rmhg krb5kdc[13656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253142, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:42.608639 15816 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:25:42.610318 15816 webserver.cc:460] Webserver started at http://127.22.1.67:41555/ using document root <none> and password file <none>
I20260809 05:25:42.611021 15816 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:42.611147 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:42.616703 15816 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260809 05:25:42.619557 15835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.620887 15816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:25:42.621052 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "a7bab1fa28d44acea582accfe6177f67"
format_stamp: "Formatted at 2026-08-09 05:24:56 on dist-test-slave-rmhg"
server_key: "23797552404c17cc5caae0ac0078d52e"
server_key_iv: "925f99979d90d762939aef225de132bd"
server_key_version: "encryptionkey@0"
I20260809 05:25:42.622087 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:25:42.650081 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:25:42.654549 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:25:42.654821 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:25:42.655947 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:25:42.657394 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:25:42.657495 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.657605 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:25:42.657676 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:42.688923 15816 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:44011
I20260809 05:25:42.689009 16002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:44011 every 8 connection(s)
I20260809 05:25:42.690459 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:25:42.695603 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.693590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:52735)
0809 05:25:42.694403 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.694646 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
W20260809 05:25:42.696316 16003 heartbeater.cc:644] Failed to heartbeat to 127.22.1.126:38883 (0 consecutive failures): Network error: Failed to ping master at 127.22.1.126:38883: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
I20260809 05:25:42.696744 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 15816
I20260809 05:25:42.697393 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.696961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:41443)
0809 05:25:42.697142 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.697259 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 05:25:42.698274 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.697916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:60075)
0809 05:25:42.698055 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.698165 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 05:25:42.698536 16003 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260809 05:25:42.736644 16011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:42.735814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:40315)
0809 05:25:42.736314 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:42.736409 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":246,"threads_started":1}
I20260809 05:25:43.208389 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.207506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:45015)
0809 05:25:43.208053 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:43.208150 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":246,"threads_started":1}
I20260809 05:25:43.536214 16013 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:43.536527 16013 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:43.538208 16013 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
Aug 09 05:25:43 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253141, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
Aug 09 05:25:43 dist-test-slave-rmhg krb5kdc[13656](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253141, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:25:43.562935 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.539570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:43635 (local address 127.22.1.66:46781)
0809 05:25:43.540187 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:43.540203 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:43.540249 (+ 46us) server_negotiation.cc:407] Connection header received
0809 05:25:43.542304 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:43.542314 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:43.542681 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:43.543060 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:43.545839 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.547727 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.549975 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.551002 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.551201 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:43.556127 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:43.556136 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:43.556149 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:43.556188 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:43.559069 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:43.559962 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:43.559969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:43.559974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:43.560148 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:43.560619 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:43.560623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:43.560626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:43.561179 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:43.561360 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:43.562152 (+ 792us) server_negotiation.cc:299] Negotiation successful
0809 05:25:43.562555 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":249,"threads_started":1}
I20260809 05:25:43.562960 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.538428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:44011 (local address 127.22.1.65:59085)
0809 05:25:43.538991 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:43.539029 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:43.539632 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:43.543231 (+ 3599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:43.543243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:43.544450 (+ 1207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:25:43.545609 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:43.545639 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.547966 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.547971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:43.549778 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:43.549795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.551277 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.551282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:43.551640 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:43.552554 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:25:43.552575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:25:43.555496 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:25:43.558944 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:43.558955 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:43.558971 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:43.559622 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:43.560286 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:43.560290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:43.560293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:43.560490 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:43.561334 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:25:43.561339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:25:43.561647 (+ 308us) client_negotiation.cc:769] Sending connection context
0809 05:25:43.561889 (+ 242us) client_negotiation.cc:240] Negotiation successful
0809 05:25:43.562465 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":280,"spinlock_wait_cycles":8192,"thread_start_us":262,"threads_started":1}
I20260809 05:25:43.562958 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.538696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:59085 (local address 127.22.1.67:44011)
0809 05:25:43.539229 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:43.539246 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:43.539295 (+ 49us) server_negotiation.cc:407] Connection header received
0809 05:25:43.542145 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:43.542158 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:43.542682 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:43.543060 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:43.545831 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.547727 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.549942 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.551002 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.551201 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:43.555827 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:43.555836 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:43.555853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:43.555894 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:43.558787 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:43.559774 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:43.559783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:43.559791 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:43.560028 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:43.560613 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:43.560619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:43.560622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:43.561199 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:43.561387 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:43.562161 (+ 774us) server_negotiation.cc:299] Negotiation successful
0809 05:25:43.562540 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260809 05:25:43.563066 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.539156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:46781 (local address 127.22.1.65:43635)
0809 05:25:43.539772 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:43.539792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:25:43.539922 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:25:43.543231 (+ 3309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:25:43.543243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:25:43.544450 (+ 1207us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:25:43.545609 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:43.545640 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.547983 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.547989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:43.549783 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:25:43.549798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:25:43.551260 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:25:43.551268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:25:43.551637 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:43.552517 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:25:43.552568 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:25:43.555844 (+ 3276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:25:43.559213 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:43.559219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:43.559223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:43.559798 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:43.560270 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:25:43.560275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:25:43.560279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:25:43.560498 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:25:43.561313 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:25:43.561322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:25:43.561647 (+ 325us) client_negotiation.cc:769] Sending connection context
0809 05:25:43.561888 (+ 241us) client_negotiation.cc:240] Negotiation successful
0809 05:25:43.562658 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":469,"spinlock_wait_cycles":47360,"thread_start_us":296,"threads_started":1}
W20260809 05:25:43.567122 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:43.567504 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:43.567648 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:43.568068 16013 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:43.699967 16020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.699098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:51657)
0809 05:25:43.699593 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:43.699736 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":221,"threads_started":1}
I20260809 05:25:43.738102 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:43.737623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:34719)
0809 05:25:43.737824 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:43.737929 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 05:25:44.210307 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:44.209488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:33675)
0809 05:25:44.210010 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:44.210116 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":275,"threads_started":1}
I20260809 05:25:44.701884 16022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:44.700956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:59241)
0809 05:25:44.701543 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:44.701666 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260809 05:25:44.739830 16024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:44.739003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:39921)
0809 05:25:44.739540 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:44.739637 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":243,"threads_started":1}
I20260809 05:25:45.212129 16025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:45.211316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:44691)
0809 05:25:45.211803 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:45.211892 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260809 05:25:45.279805 16026 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:45.279996 16026 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:45.280578 16026 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
W20260809 05:25:45.281673 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:45.281879 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:45.281965 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:45.282183 16026 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:45.703729 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:45.702885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:54613)
0809 05:25:45.703424 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:45.703537 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":250,"threads_started":1}
I20260809 05:25:45.741755 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:45.740844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:59361)
0809 05:25:45.741407 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:45.741546 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":260,"threads_started":1}
I20260809 05:25:46.214095 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:46.213259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:38705)
0809 05:25:46.213795 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:46.213894 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":268,"threads_started":1}
I20260809 05:25:46.705734 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:46.704820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:41409)
0809 05:25:46.705421 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:46.705531 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":352,"threads_started":1}
I20260809 05:25:46.743620 16036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:46.742847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:34807)
0809 05:25:46.743333 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:46.743422 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
I20260809 05:25:47.215874 16037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:47.215107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:47033)
0809 05:25:47.215630 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:47.215716 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":239,"threads_started":1}
I20260809 05:25:47.707372 16039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:47.706654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:40959)
0809 05:25:47.707129 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:47.707216 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260809 05:25:47.745322 16040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:47.744527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:57853)
0809 05:25:47.745017 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:47.745108 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260809 05:25:47.749899 16041 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:47.750094 16041 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:47.750769 16041 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
W20260809 05:25:47.751626 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:47.751822 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:47.751924 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:47.752156 16041 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:48.217968 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:48.216968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:33693)
0809 05:25:48.217632 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:48.217749 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":320,"threads_started":1}
I20260809 05:25:48.709015 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:48.708276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:35007)
0809 05:25:48.708756 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:48.708837 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":269,"threads_started":1}
I20260809 05:25:48.747051 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:48.746276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:49441)
0809 05:25:48.746746 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:48.746849 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
I20260809 05:25:49.219748 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:49.218887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:59163)
0809 05:25:49.219443 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:49.219550 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":238,"threads_started":1}
I20260809 05:25:49.710831 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:49.709965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:52513)
0809 05:25:49.710546 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:49.710629 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":277,"threads_started":1}
I20260809 05:25:49.748793 16047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:49.747946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:56005)
0809 05:25:49.748448 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:49.748599 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":239,"threads_started":1}
I20260809 05:25:50.221623 16050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:50.220770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:34311)
0809 05:25:50.221245 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:50.221383 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260809 05:25:50.712574 16051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:50.711790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:39459)
0809 05:25:50.712276 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:50.712365 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260809 05:25:50.750535 16052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:50.749746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:37349)
0809 05:25:50.750204 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:50.750307 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":227,"threads_started":1}
I20260809 05:25:51.057296 16053 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:51.057562 16053 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:51.058207 16053 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
W20260809 05:25:51.059101 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:51.059319 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:51.059403 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:51.059687 16053 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:51.223351 16055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:51.222531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:55693)
0809 05:25:51.223090 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:51.223190 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":259,"threads_started":1}
I20260809 05:25:51.714378 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:51.713576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:36959)
0809 05:25:51.714103 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:51.714207 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":233,"threads_started":1}
I20260809 05:25:51.752393 16058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:51.751619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:56537)
0809 05:25:51.752127 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:51.752216 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":269,"threads_started":1}
I20260809 05:25:52.224980 16059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:52.224262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:37033)
0809 05:25:52.224711 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:52.224801 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":248,"threads_started":1}
I20260809 05:25:52.716210 16060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:52.715426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:54463)
0809 05:25:52.715891 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:52.715982 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":248,"threads_started":1}
I20260809 05:25:52.754227 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:52.753419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:60919)
0809 05:25:52.753944 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:52.754041 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":270,"threads_started":1}
I20260809 05:25:52.919292 16062 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:52.919508 16062 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:52.920204 16062 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
W20260809 05:25:52.921335 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:52.921573 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:52.921705 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:52.922003 16062 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:53.226986 16063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:53.226098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:58005)
0809 05:25:53.226696 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:53.226804 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":287,"threads_started":1}
I20260809 05:25:53.718005 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:53.717223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:59775)
0809 05:25:53.717752 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:53.717844 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":274,"threads_started":1}
I20260809 05:25:53.756011 16071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:53.755181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:46913)
0809 05:25:53.755681 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:53.755801 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260809 05:25:54.228696 16074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:54.227897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:57847)
0809 05:25:54.228365 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:54.228499 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260809 05:25:54.719669 16075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:54.718937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:52617)
0809 05:25:54.719408 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:54.719497 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":256,"threads_started":1}
I20260809 05:25:54.757928 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:54.757071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:51283)
0809 05:25:54.757648 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:54.757736 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":283,"threads_started":1}
I20260809 05:25:55.230484 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:55.229658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:50089)
0809 05:25:55.230209 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:55.230307 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":258,"threads_started":1}
I20260809 05:25:55.721515 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:55.720648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:44961)
0809 05:25:55.721228 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:55.721331 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":283,"threads_started":1}
I20260809 05:25:55.759850 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:55.758992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:38443)
0809 05:25:55.759533 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:55.759628 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":263,"threads_started":1}
I20260809 05:25:56.232141 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:56.231393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:57227)
0809 05:25:56.231852 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:56.231937 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
I20260809 05:25:56.412837 16085 raft_consensus.cc:482] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:25:56.413066 16085 raft_consensus.cc:504] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "981d9ef40a654fe283363b75c5db3898" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 37507 } } peers { permanent_uuid: "a7bab1fa28d44acea582accfe6177f67" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 44011 } } peers { permanent_uuid: "0aa57b5f27064426afb0bc9e11915869" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 46781 } }
I20260809 05:25:56.413808 16085 leader_election.cc:288] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011), 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781)
W20260809 05:25:56.414921 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a7bab1fa28d44acea582accfe6177f67 (127.22.1.67:44011): Not found: Tablet not found: c5b486373730421193aae243eab996af
W20260809 05:25:56.415159 15462 leader_election.cc:341] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0aa57b5f27064426afb0bc9e11915869 (127.22.1.66:46781): Not found: Tablet not found: c5b486373730421193aae243eab996af
I20260809 05:25:56.415256 15462 leader_election.cc:302] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 981d9ef40a654fe283363b75c5db3898; no voters: 0aa57b5f27064426afb0bc9e11915869, a7bab1fa28d44acea582accfe6177f67
I20260809 05:25:56.415529 16085 raft_consensus.cc:2726] T c5b486373730421193aae243eab996af P 981d9ef40a654fe283363b75c5db3898 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260809 05:25:56.723341 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:56.722508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:43851)
0809 05:25:56.723056 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:56.723142 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":280,"threads_started":1}
I20260809 05:25:56.762091 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:56.761231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:58947)
0809 05:25:56.761740 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:56.761867 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":240,"threads_started":1}
I20260809 05:25:57.234020 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.233296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.66:46873)
0809 05:25:57.233764 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:57.233848 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":224,"threads_started":1}
I20260809 05:25:57.705791 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 15386
I20260809 05:25:57.725879 16090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.707074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.22.1.65:37507)
0809 05:25:57.707620 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:57.707627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:57.707658 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:25:57.707733 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:57.707740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:57.707958 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:57.708096 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:57.709539 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.710485 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.711734 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.712298 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.712402 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:57.719039 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:57.719055 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:57.719074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:57.719113 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:57.722258 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.723012 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.723024 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.723026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.723168 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.723622 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.723626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.723628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.724142 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:57.724476 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:57.724918 (+ 442us) server_negotiation.cc:299] Negotiation successful
0809 05:25:57.725146 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260809 05:25:57.725909 16092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.724680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.67:35985)
0809 05:25:57.725596 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:57.725740 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":653,"threads_started":1}
I20260809 05:25:57.727322 15612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:57.817102 16094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.815989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:38883 (local address 127.22.1.65:43727)
0809 05:25:57.816576 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0809 05:25:57.816721 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.126:38883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":269,"threads_started":1}
I20260809 05:25:57.817088 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 15386
I20260809 05:25:57.839334 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 15624
I20260809 05:25:57.852995 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.840141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.22.1.66:46781)
0809 05:25:57.840604 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:57.840608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:57.840623 (+ 15us) server_negotiation.cc:407] Connection header received
0809 05:25:57.840681 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:57.840684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:57.840745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:57.840839 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:57.842317 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.843089 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.844396 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.844850 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.844964 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:57.847546 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:57.847551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:57.847554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:57.847590 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:57.850111 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.850876 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.850880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.850883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.851038 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.851433 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.851437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.851439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.851742 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:57.852016 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:57.852352 (+ 336us) server_negotiation.cc:299] Negotiation successful
0809 05:25:57.852599 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260809 05:25:57.853864 15802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:57.937131 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 15624
I20260809 05:25:57.959398 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 15816
I20260809 05:25:57.972973 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:25:57.960223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.22.1.67:44011)
0809 05:25:57.960742 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 05:25:57.960747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:25:57.960764 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:25:57.960822 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:25:57.960826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:25:57.960896 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:25:57.960985 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:25:57.962356 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.963153 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.964385 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:25:57.964836 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:25:57.964933 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:25:57.967792 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:25:57.967798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:25:57.967801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:25:57.967835 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:25:57.970427 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.971152 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.971157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.971160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.971312 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:25:57.971618 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:25:57.971621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:25:57.971623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:25:57.971896 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:25:57.972115 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0809 05:25:57.972414 (+ 299us) server_negotiation.cc:299] Negotiation successful
0809 05:25:57.972622 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":250,"threads_started":1}
I20260809 05:25:57.973896 15998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:25:58.058516 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 15816
2026-08-09T05:25:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64720 ms)
[----------] 30 tests from SecurityITest (304027 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:25:58 dist-test-slave-rmhg krb5kdc[16111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:25:58 dist-test-slave-rmhg krb5kdc[16111](info): set up 2 sockets
Aug 09 05:25:58 dist-test-slave-rmhg krb5kdc[16111](info): commencing operation
krb5kdc: starting...
W20260809 05:25:59.291954 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:25:59 dist-test-slave-rmhg krb5kdc[16111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:25:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:25:59.594858 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44387
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:44387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:25:59.932844 16133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:25:59.933195 16133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:25:59.933322 16133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:25:59.940619 16133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:25:59.940739 16133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:25:59.940784 16133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:25:59.940840 16133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:25:59.940884 16133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:25:59.950865 16133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:44387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:25:59.952171 16133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:25:59.953536 16133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:25:59.965565 16144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:59.965600 16146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:25:59.965620 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:25:59.966063 16133 server_base.cc:1034] running on GCE node
I20260809 05:25:59.966961 16133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:25:59.968357 16133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:25:59.969594 16133 hybrid_clock.cc:648] HybridClock initialized: now 1786253159969536 us; error 44 us; skew 500 ppm
I20260809 05:25:59.969942 16133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:25:59 dist-test-slave-rmhg krb5kdc[16111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:25:59.973060 16133 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:25:59.974692 16133 webserver.cc:460] Webserver started at http://127.22.1.126:37061/ using document root <none> and password file <none>
I20260809 05:25:59.975440 16133 fs_manager.cc:359] Metadata directory not provided
I20260809 05:25:59.975575 16133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:25:59.975898 16133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:25:59.979117 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2"
format_stamp: "Formatted at 2026-08-09 05:25:59 on dist-test-slave-rmhg"
server_key: "e87b9cba7d5b16ca7fabb80e4bef413e"
server_key_iv: "cf172402e7ad092a725ad270092704cc"
server_key_version: "encryptionkey@0"
I20260809 05:25:59.979874 16133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2"
format_stamp: "Formatted at 2026-08-09 05:25:59 on dist-test-slave-rmhg"
server_key: "e87b9cba7d5b16ca7fabb80e4bef413e"
server_key_iv: "cf172402e7ad092a725ad270092704cc"
server_key_version: "encryptionkey@0"
I20260809 05:25:59.985428 16133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:25:59.988873 16157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:25:59.990249 16133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:25:59.990410 16133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2"
format_stamp: "Formatted at 2026-08-09 05:25:59 on dist-test-slave-rmhg"
server_key: "e87b9cba7d5b16ca7fabb80e4bef413e"
server_key_iv: "cf172402e7ad092a725ad270092704cc"
server_key_version: "encryptionkey@0"
I20260809 05:25:59.990602 16133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:00.003700 16133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:00.007778 16133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:00.008046 16133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:00.025820 16133 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:44387
I20260809 05:26:00.025863 16236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:44387 every 8 connection(s)
I20260809 05:26:00.027232 16133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:00.031016 16237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:00.034374 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 16133
I20260809 05:26:00.034574 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:00.035246 22533 external_mini_cluster.cc:1428] Setting key c251b69057713ce05581922461c56b14
I20260809 05:26:00.044348 16237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: Bootstrap starting.
I20260809 05:26:00.047981 16237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:00.048979 16237 log.cc:824] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:26:00 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:00.052280 16237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: No bootstrap required, opened a new log
I20260809 05:26:00.059682 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:00.037252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.22.1.126:44387)
0809 05:26:00.038095 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:00.038111 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:00.038165 (+ 54us) server_negotiation.cc:407] Connection header received
0809 05:26:00.040040 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:00.040073 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:00.040657 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:00.041118 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:00.042561 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.044562 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.046043 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.047060 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.047248 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:00.051716 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:00.051724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:00.051741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:00.051781 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:00.055036 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:00.056005 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:00.056013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:00.056021 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:00.056232 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:00.056645 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:00.056648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:00.056650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:00.057340 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:00.057582 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:00.058081 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 05:26:00.058497 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":349,"threads_started":1}
I20260809 05:26:00.061335 16237 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } }
I20260809 05:26:00.061803 16237 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:00.061918 16237 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33dcb3fae4e46b68e3bb4fcc7835ba2, State: Initialized, Role: FOLLOWER
I20260809 05:26:00.062577 16237 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } }
I20260809 05:26:00.062817 16237 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:00.062973 16237 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:00.063122 16237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:00.065516 16237 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } }
I20260809 05:26:00.065994 16237 leader_election.cc:302] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33dcb3fae4e46b68e3bb4fcc7835ba2; no voters:
I20260809 05:26:00.066565 16237 leader_election.cc:288] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:00.066690 16247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:00.067603 16247 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [term 1 LEADER]: Becoming Leader. State: Replica: b33dcb3fae4e46b68e3bb4fcc7835ba2, State: Running, Role: LEADER
I20260809 05:26:00.068163 16247 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } }
I20260809 05:26:00.068728 16237 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:00.070914 16249 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } } }
I20260809 05:26:00.071156 16250 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33dcb3fae4e46b68e3bb4fcc7835ba2. Latest consensus state: current_term: 1 leader_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33dcb3fae4e46b68e3bb4fcc7835ba2" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 44387 } } }
I20260809 05:26:00.071343 16249 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:00.071396 16250 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:00.073076 16259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:00.077927 16259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:00.086138 16259 catalog_manager.cc:1347] Generated new cluster ID: bc0cb6df6935444f8b85ac5db2bce86c
I20260809 05:26:00.086257 16259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:00.098224 16259 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:00.099973 16259 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:00.105834 16259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: Generated new TSK 0
I20260809 05:26:00.106689 16259 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:26:00.175066 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:00.524147 16279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:00.524474 16279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:00.524585 16279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:00.532122 16279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:00.532272 16279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:00.532392 16279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:00.542821 16279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:00.544313 16279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:00.545797 16279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:00.559875 16285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:00.559909 16286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:00.560173 16292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:00.560755 16279 server_base.cc:1034] running on GCE node
I20260809 05:26:00.561476 16279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:00.562479 16279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:00.563706 16279 hybrid_clock.cc:648] HybridClock initialized: now 1786253160563637 us; error 50 us; skew 500 ppm
I20260809 05:26:00.564033 16279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:00 dist-test-slave-rmhg krb5kdc[16111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:00.567248 16279 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:26:00.568760 16279 webserver.cc:460] Webserver started at http://127.22.1.65:46479/ using document root <none> and password file <none>
I20260809 05:26:00.569572 16279 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:00.569685 16279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:00.570079 16279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:00.573575 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "c68cfa55144b4d6db8d3d9c23bafc778"
format_stamp: "Formatted at 2026-08-09 05:26:00 on dist-test-slave-rmhg"
server_key: "417c04f8004aa5326e9695d12bf50ffd"
server_key_iv: "96ac5634881fcf3861a49d615d062ae1"
server_key_version: "encryptionkey@0"
I20260809 05:26:00.574546 16279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "c68cfa55144b4d6db8d3d9c23bafc778"
format_stamp: "Formatted at 2026-08-09 05:26:00 on dist-test-slave-rmhg"
server_key: "417c04f8004aa5326e9695d12bf50ffd"
server_key_iv: "96ac5634881fcf3861a49d615d062ae1"
server_key_version: "encryptionkey@0"
I20260809 05:26:00.580449 16279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:26:00.584401 16301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:00.585988 16279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:00.586170 16279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "c68cfa55144b4d6db8d3d9c23bafc778"
format_stamp: "Formatted at 2026-08-09 05:26:00 on dist-test-slave-rmhg"
server_key: "417c04f8004aa5326e9695d12bf50ffd"
server_key_iv: "96ac5634881fcf3861a49d615d062ae1"
server_key_version: "encryptionkey@0"
I20260809 05:26:00.586403 16279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:00.621474 16279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:00.625711 16279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:00.625986 16279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:00.626981 16279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:00.628374 16279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:00.628448 16279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:00.628584 16279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:00.628647 16279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:00.660117 16279 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:45565
I20260809 05:26:00.660166 16471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:45565 every 8 connection(s)
I20260809 05:26:00.661537 16279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:00.666015 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 16279
I20260809 05:26:00.666241 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:00.666882 22533 external_mini_cluster.cc:1428] Setting key 6b562ed22a608f1844bcbffb01df25d7
Aug 09 05:26:00 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253160, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:00.681526 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:00.664221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:34777 (local address 127.22.1.126:44387)
0809 05:26:00.664688 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:00.664693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:00.665746 (+ 1053us) server_negotiation.cc:407] Connection header received
0809 05:26:00.667667 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:00.667674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:00.667782 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:00.667878 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:00.669506 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.670291 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.671829 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.672255 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.672370 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:00.675927 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:00.675932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:00.675936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:00.675970 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:00.678422 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:00.679164 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:00.679168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:00.679171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:00.679295 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:00.679726 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:00.679729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:00.679731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:00.680002 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:00.680145 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:00.680891 (+ 746us) server_negotiation.cc:299] Negotiation successful
0809 05:26:00.681144 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":229,"threads_started":1}
I20260809 05:26:00.682730 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:00.664836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44387 (local address 127.22.1.65:34777)
0809 05:26:00.665585 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:00.665630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:00.667393 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:00.668131 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:00.668140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:00.668663 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:00.669351 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:00.669370 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.670429 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.670433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:00.671693 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:00.671705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:00.672493 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:00.672501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:00.672744 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:00.673437 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:00.673465 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:00.675699 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:00.678573 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:00.678583 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:00.678599 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:00.679037 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:00.679402 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:00.679407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:00.679410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:00.679611 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:00.680137 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:00.680145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:00.680499 (+ 354us) client_negotiation.cc:769] Sending connection context
0809 05:26:00.680758 (+ 259us) client_negotiation.cc:240] Negotiation successful
0809 05:26:00.681362 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":318,"threads_started":1}
I20260809 05:26:00.684316 16472 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44387
I20260809 05:26:00.684684 16472 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:00.685631 16472 heartbeater.cc:511] Master 127.22.1.126:44387 requested a full tablet report, sending...
I20260809 05:26:00.687950 16182 ts_manager.cc:194] Registered new tserver with Master: c68cfa55144b4d6db8d3d9c23bafc778 (127.22.1.65:45565)
I20260809 05:26:00.689603 16182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:34777
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:26:00.733995 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:01.077454 16479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:01.077826 16479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:01.077981 16479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:01.085883 16479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:01.086009 16479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:01.086176 16479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:01.096581 16479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:01.098083 16479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:01.099505 16479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:01.112623 16493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:01.112711 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:01.112699 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:01.113626 16479 server_base.cc:1034] running on GCE node
I20260809 05:26:01.114235 16479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:01.115154 16479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:01.116357 16479 hybrid_clock.cc:648] HybridClock initialized: now 1786253161116308 us; error 35 us; skew 500 ppm
I20260809 05:26:01.116680 16479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:01 dist-test-slave-rmhg krb5kdc[16111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253161, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:01.119748 16479 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:26:01.121187 16479 webserver.cc:460] Webserver started at http://127.22.1.66:35247/ using document root <none> and password file <none>
I20260809 05:26:01.121882 16479 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:01.122006 16479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:01.122318 16479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:01.125424 16479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "f2f457ab3e124d6c8b2465eef25d914e"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "2045d5746f804f964dea24b9e0a38e8d"
server_key_iv: "5b332a916f948210008c5de614c33ec0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.126199 16479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "f2f457ab3e124d6c8b2465eef25d914e"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "2045d5746f804f964dea24b9e0a38e8d"
server_key_iv: "5b332a916f948210008c5de614c33ec0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.131389 16479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:01.136029 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.137699 16479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:01.137890 16479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "f2f457ab3e124d6c8b2465eef25d914e"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "2045d5746f804f964dea24b9e0a38e8d"
server_key_iv: "5b332a916f948210008c5de614c33ec0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.138141 16479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:01.153939 16479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:01.158257 16479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:01.158535 16479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:01.159494 16479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:01.160809 16479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:01.160884 16479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.161007 16479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:01.161057 16479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.190723 16479 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:36269
I20260809 05:26:01.190790 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:36269 every 8 connection(s)
I20260809 05:26:01.192075 16479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:01.194918 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 16479
I20260809 05:26:01.195127 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:01.195634 22533 external_mini_cluster.cc:1428] Setting key 0a6fff5e45aa65bc67c00e93ca89a4a7
Aug 09 05:26:01 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253161, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:01.214653 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.194596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:40397 (local address 127.22.1.126:44387)
0809 05:26:01.195107 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.195113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.196335 (+ 1222us) server_negotiation.cc:407] Connection header received
0809 05:26:01.198596 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.198600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.198665 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.198747 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:01.200811 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.201591 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.203654 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.204077 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.204169 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.208465 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:01.208471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:01.208476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:01.208515 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:01.211393 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.212142 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.212148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.212152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.212294 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.212739 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.212744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.212747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.213047 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:01.213231 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.213989 (+ 758us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.214269 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260809 05:26:01.215775 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.195224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44387 (local address 127.22.1.66:40397)
0809 05:26:01.196115 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:01.196172 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:01.198274 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:01.199022 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:01.199033 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:01.199785 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:01.200610 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.200647 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.201766 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.201771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.203518 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.203534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.204300 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.204306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.204616 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.205412 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:01.205451 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:01.208203 (+ 2752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:01.211529 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:01.211537 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:01.211551 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:01.212001 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:01.212407 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:01.212410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:01.212412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:01.212614 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:01.213204 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:01.213213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:01.213589 (+ 376us) client_negotiation.cc:769] Sending connection context
0809 05:26:01.213873 (+ 284us) client_negotiation.cc:240] Negotiation successful
0809 05:26:01.214494 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":311,"threads_started":1}
I20260809 05:26:01.217401 16672 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44387
I20260809 05:26:01.217717 16672 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:01.218513 16672 heartbeater.cc:511] Master 127.22.1.126:44387 requested a full tablet report, sending...
I20260809 05:26:01.220142 16182 ts_manager.cc:194] Registered new tserver with Master: f2f457ab3e124d6c8b2465eef25d914e (127.22.1.66:36269)
I20260809 05:26:01.221319 16182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:40397
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:26:01.260640 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:38061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:01.609323 16684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:01.609649 16684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:01.609762 16684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:01.617697 16684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:01.617846 16684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:01.617962 16684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:01.628141 16684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:38061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:44387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:01.629566 16684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:01.630885 16684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:01.644661 16691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:01.644644 16695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:01.644691 16692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:01.645591 16684 server_base.cc:1034] running on GCE node
I20260809 05:26:01.646272 16684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:01.647289 16684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:01.648561 16684 hybrid_clock.cc:648] HybridClock initialized: now 1786253161648490 us; error 63 us; skew 500 ppm
I20260809 05:26:01.648936 16684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:01 dist-test-slave-rmhg krb5kdc[16111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253161, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:01.652252 16684 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:26:01.654006 16684 webserver.cc:460] Webserver started at http://127.22.1.67:32787/ using document root <none> and password file <none>
I20260809 05:26:01.654794 16684 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:01.654940 16684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:01.655295 16684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:01.658574 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "42e135d68d7d4f86a6a668b483147c3f"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "77ecc315e77dbdeb3fa650b4c4a7da40"
server_key_iv: "57521350486cfa16c9d8538c6a7c7fc0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.659411 16684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "42e135d68d7d4f86a6a668b483147c3f"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "77ecc315e77dbdeb3fa650b4c4a7da40"
server_key_iv: "57521350486cfa16c9d8538c6a7c7fc0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.664969 16684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:01.668437 16706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.669905 16684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:01.670063 16684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "42e135d68d7d4f86a6a668b483147c3f"
format_stamp: "Formatted at 2026-08-09 05:26:01 on dist-test-slave-rmhg"
server_key: "77ecc315e77dbdeb3fa650b4c4a7da40"
server_key_iv: "57521350486cfa16c9d8538c6a7c7fc0"
server_key_version: "encryptionkey@0"
I20260809 05:26:01.670243 16684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:01.684576 16684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:01.688493 16684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:01.688791 16684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:01.689993 16684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:01.691453 16684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:01.691545 16684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.691668 16684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:01.691722 16684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:01.693463 16472 heartbeater.cc:503] Master 127.22.1.126:44387 was elected leader, sending a full tablet report...
I20260809 05:26:01.721740 16684 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:33703
I20260809 05:26:01.721788 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:33703 every 8 connection(s)
I20260809 05:26:01.723065 16684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:01.730724 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 16684
I20260809 05:26:01.730911 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:01.731361 22533 external_mini_cluster.cc:1428] Setting key 5dc6e93fcd5797c1158c7a9eee8df06a
Aug 09 05:26:01 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253161, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:01.744824 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.725767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:59421 (local address 127.22.1.126:44387)
0809 05:26:01.726221 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.726226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.727335 (+ 1109us) server_negotiation.cc:407] Connection header received
0809 05:26:01.729266 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.729272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.729367 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.729474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:01.731217 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.732055 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.734167 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.734736 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.734849 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.738838 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:01.738844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:01.738847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:01.738888 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:01.741525 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.742474 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.742479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.742481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.742626 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.743130 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.743135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.743138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.743431 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:01.743564 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.744346 (+ 782us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.744548 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260809 05:26:01.746147 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.726341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:44387 (local address 127.22.1.67:59421)
0809 05:26:01.727152 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:01.727194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:01.728940 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:01.729778 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:01.729787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:01.730355 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:01.731032 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.731053 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.732240 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.732244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.733971 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.733999 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.735009 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.735016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.735265 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.735973 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:01.735998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:01.738542 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:01.741734 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:01.741745 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:01.741766 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:01.742349 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:01.742756 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:01.742760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:01.742763 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:01.743027 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:01.743586 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:01.743593 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:01.743987 (+ 394us) client_negotiation.cc:769] Sending connection context
0809 05:26:01.744267 (+ 280us) client_negotiation.cc:240] Negotiation successful
0809 05:26:01.744792 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":324,"threads_started":1}
I20260809 05:26:01.747738 16879 heartbeater.cc:347] Connected to a master server at 127.22.1.126:44387
I20260809 05:26:01.748059 16879 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:01.748787 16879 heartbeater.cc:511] Master 127.22.1.126:44387 requested a full tablet report, sending...
I20260809 05:26:01.750285 16181 ts_manager.cc:194] Registered new tserver with Master: 42e135d68d7d4f86a6a668b483147c3f (127.22.1.67:33703)
I20260809 05:26:01.751209 16181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:59421
I20260809 05:26:01.759812 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:01.777261 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.764555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.22.1.126:44387)
0809 05:26:01.764716 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.764720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.765233 (+ 513us) server_negotiation.cc:407] Connection header received
0809 05:26:01.765399 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.765403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.765470 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.765558 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:01.766956 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.767696 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.768983 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.769442 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.769546 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.771850 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:01.771854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:01.771858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:01.771890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:01.774495 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.775281 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.775285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.775288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.775429 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:01.775845 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:01.775848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:01.775850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:01.776202 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:01.776390 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.776751 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.776980 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:26:01.781762 16181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:01.788658 16181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:01.826131 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.801783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.22.1.66:36269)
0809 05:26:01.813886 (+ 12103us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.813899 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.815641 (+ 1742us) server_negotiation.cc:407] Connection header received
0809 05:26:01.816103 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.816112 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.816382 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.816600 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:01.817782 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.819565 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.823224 (+ 3659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.824734 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.824870 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.825077 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.825580 (+ 503us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.825767 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11836,"thread_start_us":12144,"threads_started":1}
I20260809 05:26:01.826340 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.801785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33703 (local address 127.0.0.1:53100)
0809 05:26:01.815428 (+ 13643us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:01.815487 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:01.815825 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:01.817284 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:01.817290 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:01.817319 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:01.817841 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.817853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.820042 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.820049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.822710 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.822729 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.825020 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.825028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.825678 (+ 650us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.825705 (+ 27us) client_negotiation.cc:769] Sending connection context
0809 05:26:01.825780 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 05:26:01.825916 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13401,"mutex_wait_us":497,"spinlock_wait_cycles":27264,"thread_start_us":13258,"threads_started":1}
I20260809 05:26:01.826366 16897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.801565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36269 (local address 127.0.0.1:51756)
0809 05:26:01.815428 (+ 13863us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:01.815494 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:01.815826 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:01.816945 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:01.816955 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:01.816998 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:01.817531 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.817547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.819799 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.819806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.823057 (+ 3251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.823072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.824959 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.824967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.825295 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.825347 (+ 52us) client_negotiation.cc:769] Sending connection context
0809 05:26:01.825460 (+ 113us) client_negotiation.cc:240] Negotiation successful
0809 05:26:01.825714 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13579,"mutex_wait_us":988,"spinlock_wait_cycles":9984,"thread_start_us":13362,"threads_started":1}
I20260809 05:26:01.827286 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.802128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.22.1.67:33703)
0809 05:26:01.814764 (+ 12636us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.814774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.815629 (+ 855us) server_negotiation.cc:407] Connection header received
0809 05:26:01.816481 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.816489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.816733 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.816942 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:01.818063 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.819843 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.822926 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.824415 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.824590 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.824855 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.826875 (+ 2020us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.827041 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12438,"thread_start_us":12251,"threads_started":1}
I20260809 05:26:01.830214 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.816221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:45565 (local address 127.0.0.1:56218)
0809 05:26:01.821566 (+ 5345us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:01.821588 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:01.821726 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:01.822388 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:01.822393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:01.822419 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:01.822883 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.822898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.824541 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.825209 (+ 668us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.827293 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:01.827309 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.829515 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.829520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:01.829643 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.829667 (+ 24us) client_negotiation.cc:769] Sending connection context
0809 05:26:01.829748 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:26:01.829885 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5168,"thread_start_us":5009,"threads_started":1}
I20260809 05:26:01.830349 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:01.816485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.22.1.65:45565)
0809 05:26:01.821042 (+ 4557us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:01.821054 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:01.821857 (+ 803us) server_negotiation.cc:407] Connection header received
0809 05:26:01.821969 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:01.821977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:01.822198 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:01.822376 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:01.823101 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.824348 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.827523 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:01.829007 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:01.829183 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:01.829389 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:01.829828 (+ 439us) server_negotiation.cc:299] Negotiation successful
0809 05:26:01.830023 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4294,"thread_start_us":4098,"threads_started":1}
I20260809 05:26:01.831079 16577 tablet_service.cc:1467] Processing CreateTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 (DEFAULT_TABLE table=test-table [id=2ece4a525281498c9712f843c2107338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:01.831079 16786 tablet_service.cc:1467] Processing CreateTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 (DEFAULT_TABLE table=test-table [id=2ece4a525281498c9712f843c2107338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:01.832763 16577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340a4239cc5b4c74a17ccf4d39ed0982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:01.832762 16786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340a4239cc5b4c74a17ccf4d39ed0982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:01.834087 16376 tablet_service.cc:1467] Processing CreateTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 (DEFAULT_TABLE table=test-table [id=2ece4a525281498c9712f843c2107338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:01.835338 16376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340a4239cc5b4c74a17ccf4d39ed0982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:01.847293 16909 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Bootstrap starting.
I20260809 05:26:01.849592 16910 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Bootstrap starting.
I20260809 05:26:01.852821 16909 tablet_bootstrap.cc:654] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:01.853282 16910 tablet_bootstrap.cc:654] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:01.854081 16911 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Bootstrap starting.
I20260809 05:26:01.854360 16909 log.cc:824] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:01.854534 16910 log.cc:824] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:01.857468 16911 tablet_bootstrap.cc:654] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:01.858917 16911 log.cc:824] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:01.860817 16910 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: No bootstrap required, opened a new log
I20260809 05:26:01.860816 16909 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: No bootstrap required, opened a new log
I20260809 05:26:01.861181 16909 ts_tablet_manager.cc:1397] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260809 05:26:01.861181 16910 ts_tablet_manager.cc:1397] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260809 05:26:01.861804 16911 tablet_bootstrap.cc:492] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: No bootstrap required, opened a new log
I20260809 05:26:01.862073 16911 ts_tablet_manager.cc:1397] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260809 05:26:01.869732 16911 raft_consensus.cc:348] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.870187 16911 raft_consensus.cc:374] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:01.870288 16911 raft_consensus.cc:729] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e135d68d7d4f86a6a668b483147c3f, State: Initialized, Role: FOLLOWER
I20260809 05:26:01.870971 16911 consensus_queue.cc:260] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.872072 16879 heartbeater.cc:503] Master 127.22.1.126:44387 was elected leader, sending a full tablet report...
I20260809 05:26:01.871970 16909 raft_consensus.cc:348] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.871970 16910 raft_consensus.cc:348] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.872406 16909 raft_consensus.cc:374] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:01.872406 16910 raft_consensus.cc:374] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:01.872498 16910 raft_consensus.cc:729] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2f457ab3e124d6c8b2465eef25d914e, State: Initialized, Role: FOLLOWER
I20260809 05:26:01.872512 16909 raft_consensus.cc:729] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68cfa55144b4d6db8d3d9c23bafc778, State: Initialized, Role: FOLLOWER
I20260809 05:26:01.872799 16911 ts_tablet_manager.cc:1428] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260809 05:26:01.873122 16910 consensus_queue.cc:260] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.873207 16909 consensus_queue.cc:260] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:01.875268 16909 ts_tablet_manager.cc:1428] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260809 05:26:01.876456 16672 heartbeater.cc:503] Master 127.22.1.126:44387 was elected leader, sending a full tablet report...
I20260809 05:26:01.877075 16910 ts_tablet_manager.cc:1428] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
W20260809 05:26:01.913980 16473 tablet.cc:2369] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:01.944090 16673 tablet.cc:2369] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:01.974730 16881 tablet.cc:2369] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:26:02.050212 16925 raft_consensus.cc:482] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:02.050554 16925 raft_consensus.cc:504] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:02.052400 16925 leader_election.cc:288] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2f457ab3e124d6c8b2465eef25d914e (127.22.1.66:36269), 42e135d68d7d4f86a6a668b483147c3f (127.22.1.67:33703)
I20260809 05:26:02.059685 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.053012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:45217 (local address 127.22.1.67:33703)
0809 05:26:02.053309 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.053316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.054166 (+ 850us) server_negotiation.cc:407] Connection header received
0809 05:26:02.054361 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.054365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.054426 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.054504 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:02.055202 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.055948 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.057939 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.058728 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.058850 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.058972 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.059296 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.059422 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260809 05:26:02.059726 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.052950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:35243 (local address 127.22.1.66:36269)
0809 05:26:02.053214 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.053221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.054105 (+ 884us) server_negotiation.cc:407] Connection header received
0809 05:26:02.054334 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.054338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.054418 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.054503 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:02.055109 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.055839 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.057951 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.058748 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.058876 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.058992 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.059331 (+ 339us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.059473 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:26:02.059918 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.052990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:33703 (local address 127.22.1.65:45217)
0809 05:26:02.054047 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:02.054068 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:02.054212 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:02.054537 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:02.054542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:02.054639 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:02.055034 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:02.055044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.056095 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.056099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:02.057790 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:02.057803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.058962 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.058973 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:02.059110 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.059170 (+ 60us) client_negotiation.cc:769] Sending connection context
0809 05:26:02.059251 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:26:02.059392 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":32,"spinlock_wait_cycles":28544,"thread_start_us":605,"threads_started":1}
I20260809 05:26:02.059940 16926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.052942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:36269 (local address 127.22.1.65:35243)
0809 05:26:02.053976 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:02.053993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:02.054171 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:02.054539 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:02.054544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:02.054578 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:02.054959 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:02.054969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.055978 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.055984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:02.057789 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:02.057803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.058993 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.059000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:02.059181 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.059206 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 05:26:02.059284 (+ 78us) client_negotiation.cc:240] Negotiation successful
0809 05:26:02.059419 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":142,"thread_start_us":623,"threads_started":1}
I20260809 05:26:02.061201 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340a4239cc5b4c74a17ccf4d39ed0982" candidate_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f457ab3e124d6c8b2465eef25d914e" is_pre_election: true
I20260809 05:26:02.061198 16815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340a4239cc5b4c74a17ccf4d39ed0982" candidate_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e135d68d7d4f86a6a668b483147c3f" is_pre_election: true
I20260809 05:26:02.061848 16815 raft_consensus.cc:2454] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68cfa55144b4d6db8d3d9c23bafc778 in term 0.
I20260809 05:26:02.061841 16605 raft_consensus.cc:2454] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68cfa55144b4d6db8d3d9c23bafc778 in term 0.
I20260809 05:26:02.062625 16303 leader_election.cc:302] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e135d68d7d4f86a6a668b483147c3f, c68cfa55144b4d6db8d3d9c23bafc778; no voters:
I20260809 05:26:02.063035 16925 raft_consensus.cc:2781] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:02.063194 16925 raft_consensus.cc:482] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:02.063326 16925 raft_consensus.cc:3037] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:02.065749 16925 raft_consensus.cc:504] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:02.066501 16925 leader_election.cc:288] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [CANDIDATE]: Term 1 election: Requested vote from peers f2f457ab3e124d6c8b2465eef25d914e (127.22.1.66:36269), 42e135d68d7d4f86a6a668b483147c3f (127.22.1.67:33703)
I20260809 05:26:02.067014 16815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340a4239cc5b4c74a17ccf4d39ed0982" candidate_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e135d68d7d4f86a6a668b483147c3f"
I20260809 05:26:02.067003 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340a4239cc5b4c74a17ccf4d39ed0982" candidate_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2f457ab3e124d6c8b2465eef25d914e"
I20260809 05:26:02.067231 16815 raft_consensus.cc:3037] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:02.067233 16605 raft_consensus.cc:3037] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:02.069567 16605 raft_consensus.cc:2454] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68cfa55144b4d6db8d3d9c23bafc778 in term 1.
I20260809 05:26:02.069581 16815 raft_consensus.cc:2454] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68cfa55144b4d6db8d3d9c23bafc778 in term 1.
I20260809 05:26:02.069996 16303 leader_election.cc:302] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e135d68d7d4f86a6a668b483147c3f, c68cfa55144b4d6db8d3d9c23bafc778; no voters:
I20260809 05:26:02.070271 16925 raft_consensus.cc:2781] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:02.070700 16925 raft_consensus.cc:686] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 1 LEADER]: Becoming Leader. State: Replica: c68cfa55144b4d6db8d3d9c23bafc778, State: Running, Role: LEADER
I20260809 05:26:02.071247 16925 consensus_queue.cc:237] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } }
I20260809 05:26:02.075923 16182 catalog_manager.cc:5495] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 reported cstate change: term changed from 0 to 1, leader changed from <none> to c68cfa55144b4d6db8d3d9c23bafc778 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68cfa55144b4d6db8d3d9c23bafc778" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 45565 } health_report { overall_health: HEALTHY } } }
I20260809 05:26:02.137584 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.131180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.22.1.65:45565)
0809 05:26:02.131372 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.131377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.131394 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:26:02.131511 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.131514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.131595 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.131727 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:02.132116 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.132884 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.134609 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.135132 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.135300 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.136013 (+ 713us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:02.136760 (+ 747us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:02.136914 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.137168 (+ 254us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.137272 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:26:02.146446 16815 raft_consensus.cc:1261] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 1 FOLLOWER]: Refusing update from remote peer c68cfa55144b4d6db8d3d9c23bafc778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:02.147558 16928 consensus_queue.cc:1035] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e135d68d7d4f86a6a668b483147c3f" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 33703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:02.147850 16605 raft_consensus.cc:1261] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 1 FOLLOWER]: Refusing update from remote peer c68cfa55144b4d6db8d3d9c23bafc778: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:02.148648 16928 consensus_queue.cc:1035] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2f457ab3e124d6c8b2465eef25d914e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 36269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:02.174072 16182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:02.174404 16182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:02.178331 16182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b33dcb3fae4e46b68e3bb4fcc7835ba2: Sending DeleteTablet for 3 replicas of tablet 340a4239cc5b4c74a17ccf4d39ed0982
I20260809 05:26:02.179641 16577 tablet_service.cc:1514] Processing DeleteTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:02 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:51756
I20260809 05:26:02.179672 16376 tablet_service.cc:1514] Processing DeleteTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:02 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:56218
I20260809 05:26:02.179787 16786 tablet_service.cc:1514] Processing DeleteTablet for tablet 340a4239cc5b4c74a17ccf4d39ed0982 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:02 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53100
I20260809 05:26:02.181615 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 16279
I20260809 05:26:02.190661 16950 tablet_replica.cc:330] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: stopping tablet replica
I20260809 05:26:02.191219 16950 raft_consensus.cc:2229] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:02.191910 16950 raft_consensus.cc:2258] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:02.192924 16948 tablet_replica.cc:330] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: stopping tablet replica
I20260809 05:26:02.193473 16948 raft_consensus.cc:2229] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:02.194057 16948 raft_consensus.cc:2258] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:02.195147 16946 tablet_replica.cc:330] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: stopping tablet replica
I20260809 05:26:02.195627 16946 raft_consensus.cc:2229] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:26:02.196379 16946 raft_consensus.cc:2258] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:02.197088 16950 ts_tablet_manager.cc:1905] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:02.198647 16946 ts_tablet_manager.cc:1905] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:02.206964 16948 ts_tablet_manager.cc:1905] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:02.207762 16946 ts_tablet_manager.cc:1918] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:02.207959 16946 log.cc:1196] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/340a4239cc5b4c74a17ccf4d39ed0982
I20260809 05:26:02.208504 16946 ts_tablet_manager.cc:1939] T 340a4239cc5b4c74a17ccf4d39ed0982 P c68cfa55144b4d6db8d3d9c23bafc778: Deleting consensus metadata
I20260809 05:26:02.210566 16160 catalog_manager.cc:4841] TS c68cfa55144b4d6db8d3d9c23bafc778 (127.22.1.65:45565): tablet 340a4239cc5b4c74a17ccf4d39ed0982 (table test-table [id=2ece4a525281498c9712f843c2107338]) successfully deleted
I20260809 05:26:02.212764 16950 ts_tablet_manager.cc:1918] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:02.212945 16950 log.cc:1196] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/340a4239cc5b4c74a17ccf4d39ed0982
I20260809 05:26:02.213564 16950 ts_tablet_manager.cc:1939] T 340a4239cc5b4c74a17ccf4d39ed0982 P 42e135d68d7d4f86a6a668b483147c3f: Deleting consensus metadata
I20260809 05:26:02.214382 16948 ts_tablet_manager.cc:1918] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:02.214589 16948 log.cc:1196] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/340a4239cc5b4c74a17ccf4d39ed0982
I20260809 05:26:02.215162 16948 ts_tablet_manager.cc:1939] T 340a4239cc5b4c74a17ccf4d39ed0982 P f2f457ab3e124d6c8b2465eef25d914e: Deleting consensus metadata
I20260809 05:26:02.215747 16160 catalog_manager.cc:4841] TS 42e135d68d7d4f86a6a668b483147c3f (127.22.1.67:33703): tablet 340a4239cc5b4c74a17ccf4d39ed0982 (table test-table [id=2ece4a525281498c9712f843c2107338]) successfully deleted
I20260809 05:26:02.216885 16162 catalog_manager.cc:4841] TS f2f457ab3e124d6c8b2465eef25d914e (127.22.1.66:36269): tablet 340a4239cc5b4c74a17ccf4d39ed0982 (table test-table [id=2ece4a525281498c9712f843c2107338]) successfully deleted
Aug 09 05:26:02 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:26:02.225389 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.183236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.22.1.65:45565)
0809 05:26:02.183432 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.183439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.199181 (+ 15742us) server_negotiation.cc:407] Connection header received
0809 05:26:02.199526 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.199533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.199623 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.199710 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:02.204635 (+ 4925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.206011 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.209023 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.209731 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.209837 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.218447 (+ 8610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:02.218461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:02.218480 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:02.218529 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:02.221962 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.223078 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.223086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.223089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.223266 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.223741 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.223744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.223746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.224165 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:02.224411 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.224811 (+ 400us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.225028 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:26:02.226126 16464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:02.325289 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 16279
I20260809 05:26:02.351274 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 16479
Aug 09 05:26:02 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:26:02.366361 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.352119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.22.1.66:36269)
0809 05:26:02.352314 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.352321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.352340 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:26:02.352487 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.352491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.352578 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.352666 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:02.354109 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.354840 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.356146 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.356543 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.356653 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.360730 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:02.360745 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:02.360762 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:02.360798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:02.363542 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.364201 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.364210 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.364212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.364331 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.364715 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.364718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.364720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.365119 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:02.365537 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.365822 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.366039 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 05:26:02.367067 16665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:02.456856 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 16479
I20260809 05:26:02.482498 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 16684
Aug 09 05:26:02 dist-test-slave-rmhg krb5kdc[16111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:26:02.498279 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:02.483353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.22.1.67:33703)
0809 05:26:02.483557 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:02.483564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:02.483585 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:02.483747 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:02.483752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:02.483819 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:02.483920 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:02.485296 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.486024 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.487326 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:02.487712 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:02.487821 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:02.492267 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:02.492284 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:02.492304 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:02.492342 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:02.495250 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.496109 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.496117 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.496120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.496275 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:02.496678 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:02.496682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:02.496684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:02.497056 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:02.497385 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:02.497709 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 05:26:02.497931 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:26:02.498943 16871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:02.588776 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 16684
I20260809 05:26:02.612761 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 16133
I20260809 05:26:02.616588 16228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:02.741032 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 16133
2026-08-09T05:26:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4689 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-09T05:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:02Z Disabled control of system clock
I20260809 05:26:02.840229 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:37491
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:37491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:03.184636 16961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:03.184944 16961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:03.185058 16961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:03.192279 16961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:03.192409 16961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:03.192447 16961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:03.192507 16961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:03.192584 16961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:03.202011 16961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:37491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:37491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:03.203387 16961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:03.205279 16961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:03.216764 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:03.216982 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:03.217031 16966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:03.217875 16961 server_base.cc:1034] running on GCE node
I20260809 05:26:03.218775 16961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:03.220291 16961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:03.221493 16961 hybrid_clock.cc:648] HybridClock initialized: now 1786253163221437 us; error 44 us; skew 500 ppm
I20260809 05:26:03.221853 16961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:03.227183 16961 webserver.cc:460] Webserver started at http://127.22.1.126:41157/ using document root <none> and password file <none>
I20260809 05:26:03.227923 16961 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:03.228031 16961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:03.228363 16961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:03.231361 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "ee096e5d00364789bc0cf0f2233f7917"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "07293c025212d1d86f1a19fbc570764c"
server_key_iv: "016cfa711272e6b94c81f41b784c20af"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.232177 16961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "ee096e5d00364789bc0cf0f2233f7917"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "07293c025212d1d86f1a19fbc570764c"
server_key_iv: "016cfa711272e6b94c81f41b784c20af"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.237334 16961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:26:03.240700 16976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:03.242072 16961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:03.242238 16961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "ee096e5d00364789bc0cf0f2233f7917"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "07293c025212d1d86f1a19fbc570764c"
server_key_iv: "016cfa711272e6b94c81f41b784c20af"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.242442 16961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:03.274470 16961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:03.275350 16961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:03.275619 16961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:03.293133 16961 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:37491
I20260809 05:26:03.293112 17050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:37491 every 8 connection(s)
I20260809 05:26:03.294646 16961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:03.298780 17051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:03.300221 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 16961
I20260809 05:26:03.300418 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:03.301023 22533 external_mini_cluster.cc:1428] Setting key 2d0316287838fbf2453033d1ef5a5c66
I20260809 05:26:03.313601 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: Bootstrap starting.
I20260809 05:26:03.315800 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:03.303358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.22.1.126:37491)
0809 05:26:03.304468 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:03.304481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:03.304524 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:26:03.306193 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:03.306224 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:03.306782 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:03.307131 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:03.307804 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.309093 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.311123 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.311853 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.311993 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:03.313363 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:03.313371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:03.313387 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:03.313980 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:03.314069 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:03.314097 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:03.314201 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:03.314450 (+ 249us) server_negotiation.cc:299] Negotiation successful
0809 05:26:03.314790 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":365,"threads_started":1}
I20260809 05:26:03.317296 17051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:03.318341 17051 log.cc:824] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:03.321175 17051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: No bootstrap required, opened a new log
I20260809 05:26:03.328429 17051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } }
I20260809 05:26:03.328814 17051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:03.328914 17051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee096e5d00364789bc0cf0f2233f7917, State: Initialized, Role: FOLLOWER
I20260809 05:26:03.329533 17051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } }
I20260809 05:26:03.329771 17051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:03.329906 17051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:03.330127 17051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:03.332427 17051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } }
I20260809 05:26:03.333000 17051 leader_election.cc:302] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee096e5d00364789bc0cf0f2233f7917; no voters:
I20260809 05:26:03.333686 17051 leader_election.cc:288] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:03.333833 17062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:03.335220 17062 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [term 1 LEADER]: Becoming Leader. State: Replica: ee096e5d00364789bc0cf0f2233f7917, State: Running, Role: LEADER
I20260809 05:26:03.335803 17062 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } }
I20260809 05:26:03.335992 17051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:03.338807 17064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee096e5d00364789bc0cf0f2233f7917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } } }
I20260809 05:26:03.338804 17065 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee096e5d00364789bc0cf0f2233f7917. Latest consensus state: current_term: 1 leader_uuid: "ee096e5d00364789bc0cf0f2233f7917" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee096e5d00364789bc0cf0f2233f7917" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 37491 } } }
I20260809 05:26:03.339357 17064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:03.339378 17065 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:03.341053 17075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:03.346645 17075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:03.356503 17075 catalog_manager.cc:1347] Generated new cluster ID: 4ac92de7591647f696b3c11d2da53411
I20260809 05:26:03.356640 17075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:03.369522 17075 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:03.371197 17075 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:03.387223 17075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: Generated new TSK 0
I20260809 05:26:03.388367 17075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:03.401113 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--builtin_ntp_servers=127.22.1.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:03.752332 17091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:03.752632 17091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:03.752722 17091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:03.760071 17091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:03.760182 17091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:03.760293 17091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:03.769534 17091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:03.770778 17091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:03.772533 17091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:03.786265 17099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:03.786293 17104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:03.786211 17101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:03.787222 17091 server_base.cc:1034] running on GCE node
I20260809 05:26:03.787858 17091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:03.788858 17091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:03.790064 17091 hybrid_clock.cc:648] HybridClock initialized: now 1786253163790010 us; error 42 us; skew 500 ppm
I20260809 05:26:03.790392 17091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:03.792623 17091 webserver.cc:460] Webserver started at http://127.22.1.65:36333/ using document root <none> and password file <none>
I20260809 05:26:03.793350 17091 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:03.793457 17091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:03.793756 17091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:03.796808 17091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "bf596b1abd384c82a70bdb116468497b"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "c295b5035884373393b93427e80637a8"
server_key_iv: "d0c4f75e2ae67f3a8344c0d19714c3e4"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.797633 17091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "bf596b1abd384c82a70bdb116468497b"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "c295b5035884373393b93427e80637a8"
server_key_iv: "d0c4f75e2ae67f3a8344c0d19714c3e4"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.802805 17091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:26:03.806277 17113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:03.807552 17091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:26:03.807709 17091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "bf596b1abd384c82a70bdb116468497b"
format_stamp: "Formatted at 2026-08-09 05:26:03 on dist-test-slave-rmhg"
server_key: "c295b5035884373393b93427e80637a8"
server_key_iv: "d0c4f75e2ae67f3a8344c0d19714c3e4"
server_key_version: "encryptionkey@0"
I20260809 05:26:03.807899 17091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:03.832093 17091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:03.832966 17091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:03.833292 17091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:03.834368 17091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:03.836072 17091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:03.836184 17091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:03.836289 17091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:03.836361 17091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:03.867473 17091 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:41067
I20260809 05:26:03.867597 17298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:41067 every 8 connection(s)
I20260809 05:26:03.868778 17091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:03.871095 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 17091
I20260809 05:26:03.871286 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:03.871868 22533 external_mini_cluster.cc:1428] Setting key e8bf9f2972ae1d19b9931e0dc22c1d82
I20260809 05:26:03.874923 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--builtin_ntp_servers=127.22.1.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:03.882084 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:03.871430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:48157 (local address 127.22.1.126:37491)
0809 05:26:03.871891 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:03.871896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:03.873396 (+ 1500us) server_negotiation.cc:407] Connection header received
0809 05:26:03.875552 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:03.875557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:03.875638 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:03.875746 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:03.876793 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.877828 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.879375 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.879875 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.879995 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:03.881289 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:03.881296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:03.881300 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:03.881323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:03.881372 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:03.881387 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:03.881469 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:03.881657 (+ 188us) server_negotiation.cc:299] Negotiation successful
0809 05:26:03.881751 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260809 05:26:03.883338 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:03.872099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:37491 (local address 127.22.1.65:48157)
0809 05:26:03.873234 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:03.873285 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:03.875285 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:03.875935 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:03.875955 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:03.876055 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:03.876628 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:03.876645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.877981 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.877986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:03.879236 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:03.879248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:03.880077 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:03.880082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:03.880281 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:03.880952 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:03.880955 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:03.880996 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:03.881083 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:03.881473 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:03.881481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:03.881487 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:26:03.881627 (+ 140us) client_negotiation.cc:240] Negotiation successful
0809 05:26:03.881934 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":319,"threads_started":1}
I20260809 05:26:03.884896 17299 heartbeater.cc:347] Connected to a master server at 127.22.1.126:37491
I20260809 05:26:03.885250 17299 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:03.886152 17299 heartbeater.cc:511] Master 127.22.1.126:37491 requested a full tablet report, sending...
I20260809 05:26:03.888314 16998 ts_manager.cc:194] Registered new tserver with Master: bf596b1abd384c82a70bdb116468497b (127.22.1.65:41067)
I20260809 05:26:03.889797 16998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.65:48157
W20260809 05:26:04.225008 17306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:04.225376 17306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:04.225522 17306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:04.232704 17306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:04.232818 17306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:04.232978 17306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:04.242115 17306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:04.243432 17306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:04.245410 17306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:04.258708 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:04.258720 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:04.258769 17321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:04.259599 17306 server_base.cc:1034] running on GCE node
I20260809 05:26:04.260293 17306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:04.261315 17306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:04.262526 17306 hybrid_clock.cc:648] HybridClock initialized: now 1786253164262464 us; error 50 us; skew 500 ppm
I20260809 05:26:04.262861 17306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:04.265317 17306 webserver.cc:460] Webserver started at http://127.22.1.66:43447/ using document root <none> and password file <none>
I20260809 05:26:04.266029 17306 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:04.266124 17306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:04.266436 17306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:04.269477 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "6b6d28725f4d48059e522dc5947383e4"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "c5b8532dec7e73c6fe27290c8221224b"
server_key_iv: "d29d58faf4cf42602bbbac2b57c20b9b"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.270295 17306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "6b6d28725f4d48059e522dc5947383e4"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "c5b8532dec7e73c6fe27290c8221224b"
server_key_iv: "d29d58faf4cf42602bbbac2b57c20b9b"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.275748 17306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:04.279269 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.280612 17306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:04.280777 17306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "6b6d28725f4d48059e522dc5947383e4"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "c5b8532dec7e73c6fe27290c8221224b"
server_key_iv: "d29d58faf4cf42602bbbac2b57c20b9b"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.280975 17306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:04.294998 17306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:04.295873 17306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:04.296160 17306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:04.301555 17306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:04.303185 17306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:04.303278 17306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.303390 17306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:04.303459 17306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.332957 17306 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:41969
I20260809 05:26:04.333012 17509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:41969 every 8 connection(s)
I20260809 05:26:04.334403 17306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:04.335829 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 17306
I20260809 05:26:04.336040 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:04.336555 22533 external_mini_cluster.cc:1428] Setting key ef927907c65459ecd40d0326a80b0861
I20260809 05:26:04.339715 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--builtin_ntp_servers=127.22.1.84:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:04.348234 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.337437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:57295 (local address 127.22.1.126:37491)
0809 05:26:04.337637 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.337646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.339484 (+ 1838us) server_negotiation.cc:407] Connection header received
0809 05:26:04.341377 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.341383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.341467 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.341567 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:04.342697 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.343630 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.345422 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.345994 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.346124 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.347471 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:04.347477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:04.347482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:04.347503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:04.347544 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:04.347559 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:04.347644 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.347831 (+ 187us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.347933 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:26:04.349229 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.338130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:37491 (local address 127.22.1.66:57295)
0809 05:26:04.339271 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.339322 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.341057 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.341809 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.341817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.341917 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:04.342519 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.342536 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.343786 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.343792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.345268 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.345282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.346199 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.346204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.346407 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.347152 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:04.347155 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:04.347198 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:04.347281 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:04.347654 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:04.347661 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:04.347666 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.347804 (+ 138us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.348109 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":345,"threads_started":1}
I20260809 05:26:04.350667 17510 heartbeater.cc:347] Connected to a master server at 127.22.1.126:37491
I20260809 05:26:04.351001 17510 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:04.351948 17510 heartbeater.cc:511] Master 127.22.1.126:37491 requested a full tablet report, sending...
I20260809 05:26:04.353492 16998 ts_manager.cc:194] Registered new tserver with Master: 6b6d28725f4d48059e522dc5947383e4 (127.22.1.66:41969)
I20260809 05:26:04.354581 16998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.66:57295
W20260809 05:26:04.681965 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:04.682253 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:04.682348 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:04.689469 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:04.689565 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:04.689679 17515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:04.698668 17515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:37491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:04.699993 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:04.701823 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:04.714362 17522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:04.714506 17526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:04.715061 17523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:04.715497 17515 server_base.cc:1034] running on GCE node
I20260809 05:26:04.716159 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:04.717123 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:04.718376 17515 hybrid_clock.cc:648] HybridClock initialized: now 1786253164718316 us; error 50 us; skew 500 ppm
I20260809 05:26:04.718696 17515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:04.720952 17515 webserver.cc:460] Webserver started at http://127.22.1.67:37385/ using document root <none> and password file <none>
I20260809 05:26:04.721678 17515 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:04.721788 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:04.722095 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:04.724974 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "fad5326f7e874285b86e3a0735ddaf04"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "7834d7100f4a93238f683066916e8902"
server_key_iv: "3802464f14579f20bf48c0a12c2a71e7"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.725804 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "fad5326f7e874285b86e3a0735ddaf04"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "7834d7100f4a93238f683066916e8902"
server_key_iv: "3802464f14579f20bf48c0a12c2a71e7"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.730919 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:04.734639 17533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.736106 17515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:04.736291 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "fad5326f7e874285b86e3a0735ddaf04"
format_stamp: "Formatted at 2026-08-09 05:26:04 on dist-test-slave-rmhg"
server_key: "7834d7100f4a93238f683066916e8902"
server_key_iv: "3802464f14579f20bf48c0a12c2a71e7"
server_key_version: "encryptionkey@0"
I20260809 05:26:04.736553 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:04.759824 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:04.760720 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:04.761001 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:04.762169 17515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:04.763707 17515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:04.763810 17515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.763924 17515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:04.763998 17515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:04.794646 17515 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:38757
I20260809 05:26:04.794731 17693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:38757 every 8 connection(s)
I20260809 05:26:04.795957 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:04.800724 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 17515
I20260809 05:26:04.800899 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:04.801374 22533 external_mini_cluster.cc:1428] Setting key 521efd3a2560b909a5421a4cbb44a328
I20260809 05:26:04.809123 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.798632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:48589 (local address 127.22.1.126:37491)
0809 05:26:04.798823 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.798828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.800348 (+ 1520us) server_negotiation.cc:407] Connection header received
0809 05:26:04.802382 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.802388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.802477 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.802586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:04.803889 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.804815 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.806480 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.806902 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.807014 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.808316 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:04.808322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:04.808325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:04.808345 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:04.808383 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:04.808395 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:04.808514 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.808743 (+ 229us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.808839 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:26:04.810216 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.799200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:37491 (local address 127.22.1.67:48589)
0809 05:26:04.800164 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.800205 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.802092 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.802868 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.802880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.803004 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:04.803713 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.803732 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.804961 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.804966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.806292 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.806305 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.807087 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.807092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.807312 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.807965 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:04.807967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:04.808009 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:04.808102 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:04.808512 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:04.808517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:04.808523 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.808667 (+ 144us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.809002 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":286,"threads_started":1}
I20260809 05:26:04.811790 17694 heartbeater.cc:347] Connected to a master server at 127.22.1.126:37491
I20260809 05:26:04.812103 17694 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:04.812853 17694 heartbeater.cc:511] Master 127.22.1.126:37491 requested a full tablet report, sending...
I20260809 05:26:04.814414 16998 ts_manager.cc:194] Registered new tserver with Master: fad5326f7e874285b86e3a0735ddaf04 (127.22.1.67:38757)
I20260809 05:26:04.815444 16998 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.67:48589
I20260809 05:26:04.817188 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:04.827469 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.821865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.22.1.126:37491)
0809 05:26:04.822101 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.822107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.822393 (+ 286us) server_negotiation.cc:407] Connection header received
0809 05:26:04.822544 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.822548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.822627 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.822711 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:04.823166 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.823870 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.825243 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.825808 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.825907 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.826866 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:04.826871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:04.826874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:04.826892 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:04.826925 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:04.826937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:04.827021 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.827103 (+ 82us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.827178 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:26:04.831305 16998 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:04.838285 16998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:04.875222 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.850110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:38757 (local address 127.0.0.1:48614)
0809 05:26:04.864321 (+ 14211us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.864386 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.864668 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.865907 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.865917 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.865956 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:04.867329 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.867348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.869375 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.869381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.872085 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.872102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.874368 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.874375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.874559 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.874592 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.874693 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.874827 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13937,"mutex_wait_us":142,"thread_start_us":16213,"threads_started":1}
I20260809 05:26:04.875687 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.850234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:41969 (local address 127.0.0.1:52832)
0809 05:26:04.865953 (+ 15719us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.865975 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.866102 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.867131 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.867136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.867162 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:04.867560 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.867572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.869889 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.869895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.872299 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.872313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.874173 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.874180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.874394 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.874453 (+ 59us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.874560 (+ 107us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.874746 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15522,"mutex_wait_us":118,"thread_start_us":15375,"threads_started":1}
I20260809 05:26:04.875761 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.850787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.22.1.66:41969)
0809 05:26:04.863694 (+ 12907us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.863704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.866455 (+ 2751us) server_negotiation.cc:407] Connection header received
0809 05:26:04.866583 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.866590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.866844 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.867048 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:04.867744 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.869729 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.872506 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.873932 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.874952 (+ 1020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.875194 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.875277 (+ 83us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.875453 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12715,"thread_start_us":13993,"threads_started":1}
I20260809 05:26:04.876662 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.850800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41067 (local address 127.0.0.1:52150)
0809 05:26:04.866628 (+ 15828us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.866650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.866857 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.867727 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.867732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.867757 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:04.868101 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.868113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.870056 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.870061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.873067 (+ 3006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.873081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.875916 (+ 2835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.875924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.876099 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.876131 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.876234 (+ 103us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.876359 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15704,"mutex_wait_us":783,"thread_start_us":13086,"threads_started":1}
I20260809 05:26:04.877017 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.850612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.22.1.67:38757)
0809 05:26:04.863224 (+ 12612us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.863236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.864754 (+ 1518us) server_negotiation.cc:407] Connection header received
0809 05:26:04.864981 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.864990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.865322 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.865533 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:04.867588 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.869216 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.872340 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.873811 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.873941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.874231 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.876481 (+ 2250us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.876675 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12370,"thread_start_us":12172,"threads_started":1}
I20260809 05:26:04.877701 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.851328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.22.1.65:41067)
0809 05:26:04.865439 (+ 14111us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.865448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.866759 (+ 1311us) server_negotiation.cc:407] Connection header received
0809 05:26:04.867048 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.867056 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.867293 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.867454 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:04.868310 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.869853 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.873312 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.874729 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.874838 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.875522 (+ 684us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.876308 (+ 786us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.877378 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13944,"thread_start_us":13838,"threads_started":1}
I20260809 05:26:04.880055 17406 tablet_service.cc:1467] Processing CreateTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 (DEFAULT_TABLE table=test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:04.880409 17600 tablet_service.cc:1467] Processing CreateTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 (DEFAULT_TABLE table=test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:04.881548 17406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e6fc164fb460bbc26ef6cee8b1e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:04.881103 17186 tablet_service.cc:1467] Processing CreateTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 (DEFAULT_TABLE table=test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:04.881955 17600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e6fc164fb460bbc26ef6cee8b1e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:04.882423 17186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1e6fc164fb460bbc26ef6cee8b1e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:04.892894 17299 heartbeater.cc:503] Master 127.22.1.126:37491 was elected leader, sending a full tablet report...
I20260809 05:26:04.893316 17719 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Bootstrap starting.
I20260809 05:26:04.896449 17720 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Bootstrap starting.
I20260809 05:26:04.897051 17719 tablet_bootstrap.cc:654] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:04.898870 17721 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Bootstrap starting.
I20260809 05:26:04.899080 17719 log.cc:824] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:04.900914 17720 tablet_bootstrap.cc:654] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:04.902139 17720 log.cc:824] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:04.902547 17721 tablet_bootstrap.cc:654] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:04.903460 17721 log.cc:824] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:04.904264 17719 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: No bootstrap required, opened a new log
I20260809 05:26:04.904585 17719 ts_tablet_manager.cc:1397] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260809 05:26:04.906260 17720 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: No bootstrap required, opened a new log
I20260809 05:26:04.906522 17720 ts_tablet_manager.cc:1397] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 05:26:04.906533 17721 tablet_bootstrap.cc:492] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: No bootstrap required, opened a new log
I20260809 05:26:04.906781 17721 ts_tablet_manager.cc:1397] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 05:26:04.914006 17720 raft_consensus.cc:348] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.914391 17720 raft_consensus.cc:374] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:04.914520 17720 raft_consensus.cc:729] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf596b1abd384c82a70bdb116468497b, State: Initialized, Role: FOLLOWER
I20260809 05:26:04.915114 17720 consensus_queue.cc:260] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.915988 17719 raft_consensus.cc:348] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.916440 17719 raft_consensus.cc:374] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:04.916559 17719 raft_consensus.cc:729] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6d28725f4d48059e522dc5947383e4, State: Initialized, Role: FOLLOWER
I20260809 05:26:04.916857 17721 raft_consensus.cc:348] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.917356 17721 raft_consensus.cc:374] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:04.917184 17719 consensus_queue.cc:260] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.917531 17721 raft_consensus.cc:729] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad5326f7e874285b86e3a0735ddaf04, State: Initialized, Role: FOLLOWER
I20260809 05:26:04.918046 17720 ts_tablet_manager.cc:1428] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260809 05:26:04.918290 17721 consensus_queue.cc:260] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.919478 17694 heartbeater.cc:503] Master 127.22.1.126:37491 was elected leader, sending a full tablet report...
I20260809 05:26:04.920046 17721 ts_tablet_manager.cc:1428] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 05:26:04.925851 17510 heartbeater.cc:503] Master 127.22.1.126:37491 was elected leader, sending a full tablet report...
I20260809 05:26:04.926492 17719 ts_tablet_manager.cc:1428] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Time spent starting tablet: real 0.022s user 0.007s sys 0.011s
I20260809 05:26:04.934442 17728 raft_consensus.cc:482] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:04.934736 17728 raft_consensus.cc:504] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.936476 17728 leader_election.cc:288] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b6d28725f4d48059e522dc5947383e4 (127.22.1.66:41969), bf596b1abd384c82a70bdb116468497b (127.22.1.65:41067)
I20260809 05:26:04.942951 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.936961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:39543 (local address 127.22.1.65:41067)
0809 05:26:04.937187 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.937193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.937214 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:04.937291 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.937296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.937367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.937457 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:04.938001 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.939034 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.941034 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.942209 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.942299 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.942373 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.942682 (+ 309us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.942773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260809 05:26:04.943076 17697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.936744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:41067 (local address 127.22.1.67:39543)
0809 05:26:04.936924 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.936944 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.937068 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.937551 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.937556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.937593 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:04.937825 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.937835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.939156 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.939162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.940883 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.940896 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.942344 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.942350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.942535 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.942571 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.942666 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.942805 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260809 05:26:04.944012 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.937296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:41969 (local address 127.22.1.67:37715)
0809 05:26:04.937995 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:04.938018 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:04.938142 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:04.938416 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:04.938421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:04.938445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:04.938794 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.938806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.940073 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.940077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.941780 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:04.941795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.943355 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.943362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:04.943530 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.943563 (+ 33us) client_negotiation.cc:769] Sending connection context
0809 05:26:04.943656 (+ 93us) client_negotiation.cc:240] Negotiation successful
0809 05:26:04.943772 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":5,"thread_start_us":372,"threads_started":1}
I20260809 05:26:04.944038 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.936819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:37715 (local address 127.22.1.66:41969)
0809 05:26:04.937036 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.937042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.938107 (+ 1065us) server_negotiation.cc:407] Connection header received
0809 05:26:04.938244 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.938248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.938316 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.938383 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:04.938939 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.939899 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.941961 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.942950 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.943078 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.943165 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.943685 (+ 520us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.943791 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260809 05:26:04.944324 17219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e6fc164fb460bbc26ef6cee8b1e01" candidate_uuid: "fad5326f7e874285b86e3a0735ddaf04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf596b1abd384c82a70bdb116468497b" is_pre_election: true
I20260809 05:26:04.944948 17219 raft_consensus.cc:2454] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad5326f7e874285b86e3a0735ddaf04 in term 0.
I20260809 05:26:04.945003 17436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e6fc164fb460bbc26ef6cee8b1e01" candidate_uuid: "fad5326f7e874285b86e3a0735ddaf04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6d28725f4d48059e522dc5947383e4" is_pre_election: true
I20260809 05:26:04.945501 17436 raft_consensus.cc:2454] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fad5326f7e874285b86e3a0735ddaf04 in term 0.
I20260809 05:26:04.945624 17534 leader_election.cc:302] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf596b1abd384c82a70bdb116468497b, fad5326f7e874285b86e3a0735ddaf04; no voters:
I20260809 05:26:04.946034 17728 raft_consensus.cc:2781] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:04.946177 17728 raft_consensus.cc:482] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:04.946276 17728 raft_consensus.cc:3037] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:04.948510 17728 raft_consensus.cc:504] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.949080 17728 leader_election.cc:288] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [CANDIDATE]: Term 1 election: Requested vote from peers 6b6d28725f4d48059e522dc5947383e4 (127.22.1.66:41969), bf596b1abd384c82a70bdb116468497b (127.22.1.65:41067)
I20260809 05:26:04.949532 17436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e6fc164fb460bbc26ef6cee8b1e01" candidate_uuid: "fad5326f7e874285b86e3a0735ddaf04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b6d28725f4d48059e522dc5947383e4"
I20260809 05:26:04.949613 17219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be1e6fc164fb460bbc26ef6cee8b1e01" candidate_uuid: "fad5326f7e874285b86e3a0735ddaf04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf596b1abd384c82a70bdb116468497b"
I20260809 05:26:04.949743 17436 raft_consensus.cc:3037] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:04.949798 17219 raft_consensus.cc:3037] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:04.951907 17436 raft_consensus.cc:2454] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad5326f7e874285b86e3a0735ddaf04 in term 1.
I20260809 05:26:04.951977 17219 raft_consensus.cc:2454] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fad5326f7e874285b86e3a0735ddaf04 in term 1.
I20260809 05:26:04.952263 17535 leader_election.cc:302] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6d28725f4d48059e522dc5947383e4, fad5326f7e874285b86e3a0735ddaf04; no voters:
I20260809 05:26:04.952540 17728 raft_consensus.cc:2781] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:04.952947 17728 raft_consensus.cc:686] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 1 LEADER]: Becoming Leader. State: Replica: fad5326f7e874285b86e3a0735ddaf04, State: Running, Role: LEADER
I20260809 05:26:04.953451 17728 consensus_queue.cc:237] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } }
I20260809 05:26:04.957828 16998 catalog_manager.cc:5495] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 reported cstate change: term changed from 0 to 1, leader changed from <none> to fad5326f7e874285b86e3a0735ddaf04 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "fad5326f7e874285b86e3a0735ddaf04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fad5326f7e874285b86e3a0735ddaf04" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 38757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 } health_report { overall_health: UNKNOWN } } }
I20260809 05:26:04.981629 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:04.976192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.22.1.67:38757)
0809 05:26:04.976374 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:04.976379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:04.976397 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:26:04.976582 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:04.976586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:04.976681 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:04.976816 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:04.977256 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.978027 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.979524 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:04.979947 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:04.980050 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:04.980555 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:04.981132 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:04.981302 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:04.981343 (+ 41us) server_negotiation.cc:299] Negotiation successful
0809 05:26:04.981419 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:26:04.989889 17436 raft_consensus.cc:1261] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 1 FOLLOWER]: Refusing update from remote peer fad5326f7e874285b86e3a0735ddaf04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:04.989890 17219 raft_consensus.cc:1261] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 1 FOLLOWER]: Refusing update from remote peer fad5326f7e874285b86e3a0735ddaf04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:04.990823 17728 consensus_queue.cc:1035] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b6d28725f4d48059e522dc5947383e4" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:04.991110 17731 consensus_queue.cc:1035] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf596b1abd384c82a70bdb116468497b" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:05.014181 16998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:05.014410 16998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:05.018309 16998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee096e5d00364789bc0cf0f2233f7917: Sending DeleteTablet for 3 replicas of tablet be1e6fc164fb460bbc26ef6cee8b1e01
I20260809 05:26:05.019845 17600 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:05 UTC) from {username='slave'} at 127.0.0.1:48614
I20260809 05:26:05.019888 17406 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:05 UTC) from {username='slave'} at 127.0.0.1:52832
I20260809 05:26:05.020968 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 17091
I20260809 05:26:05.021003 17186 tablet_service.cc:1514] Processing DeleteTablet for tablet be1e6fc164fb460bbc26ef6cee8b1e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:05 UTC) from {username='slave'} at 127.0.0.1:52150
I20260809 05:26:05.044669 17755 tablet_replica.cc:330] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: stopping tablet replica
I20260809 05:26:05.045482 17752 tablet_replica.cc:330] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: stopping tablet replica
I20260809 05:26:05.046229 17755 raft_consensus.cc:2229] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:05.047498 17755 raft_consensus.cc:2258] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:05.046861 17752 raft_consensus.cc:2229] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:26:05.048380 17752 raft_consensus.cc:2258] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:05.049604 17753 tablet_replica.cc:330] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: stopping tablet replica
I20260809 05:26:05.050104 17753 raft_consensus.cc:2229] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:05.050666 17753 raft_consensus.cc:2258] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:05.050974 17755 ts_tablet_manager.cc:1905] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:05.050990 17752 ts_tablet_manager.cc:1905] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:05.053191 17753 ts_tablet_manager.cc:1905] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:05.058544 17755 ts_tablet_manager.cc:1918] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:05.058753 17755 log.cc:1196] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/be1e6fc164fb460bbc26ef6cee8b1e01
I20260809 05:26:05.059319 17755 ts_tablet_manager.cc:1939] T be1e6fc164fb460bbc26ef6cee8b1e01 P bf596b1abd384c82a70bdb116468497b: Deleting consensus metadata
I20260809 05:26:05.060510 17753 ts_tablet_manager.cc:1918] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:05.060719 17753 log.cc:1196] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/be1e6fc164fb460bbc26ef6cee8b1e01
I20260809 05:26:05.061326 17753 ts_tablet_manager.cc:1939] T be1e6fc164fb460bbc26ef6cee8b1e01 P 6b6d28725f4d48059e522dc5947383e4: Deleting consensus metadata
I20260809 05:26:05.061450 16977 catalog_manager.cc:4841] TS bf596b1abd384c82a70bdb116468497b (127.22.1.65:41067): tablet be1e6fc164fb460bbc26ef6cee8b1e01 (table test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]) successfully deleted
I20260809 05:26:05.061679 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:05.021818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.22.1.65:41067)
0809 05:26:05.021988 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:05.021995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:05.044259 (+ 22264us) server_negotiation.cc:407] Connection header received
0809 05:26:05.044827 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:05.044833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:05.044959 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:05.045042 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:05.045925 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.047149 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.051883 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.052531 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.052641 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:05.059507 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:05.059524 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:05.059547 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:05.060513 (+ 966us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:05.060593 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:05.060629 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:05.060755 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:05.061205 (+ 450us) server_negotiation.cc:299] Negotiation successful
0809 05:26:05.061339 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:26:05.062968 17287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:05.063239 16978 catalog_manager.cc:4841] TS 6b6d28725f4d48059e522dc5947383e4 (127.22.1.66:41969): tablet be1e6fc164fb460bbc26ef6cee8b1e01 (table test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]) successfully deleted
I20260809 05:26:05.065068 17752 ts_tablet_manager.cc:1918] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:05.065348 17752 log.cc:1196] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/be1e6fc164fb460bbc26ef6cee8b1e01
I20260809 05:26:05.065941 17752 ts_tablet_manager.cc:1939] T be1e6fc164fb460bbc26ef6cee8b1e01 P fad5326f7e874285b86e3a0735ddaf04: Deleting consensus metadata
I20260809 05:26:05.067598 16980 catalog_manager.cc:4841] TS fad5326f7e874285b86e3a0735ddaf04 (127.22.1.67:38757): tablet be1e6fc164fb460bbc26ef6cee8b1e01 (table test-table [id=f7d0bc2e6ff445f3b8fd5cc9b70e584a]) successfully deleted
I20260809 05:26:05.174533 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 17091
I20260809 05:26:05.203352 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 17306
I20260809 05:26:05.210367 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:05.204161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.22.1.66:41969)
0809 05:26:05.204362 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:05.204367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:05.204387 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:26:05.204539 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:05.204545 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:05.204651 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:05.204738 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:05.205183 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.205949 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.207302 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.207722 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.207836 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:05.208899 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:05.208908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:05.208924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:05.209614 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:05.209664 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:05.209692 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:05.209793 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:05.209926 (+ 133us) server_negotiation.cc:299] Negotiation successful
0809 05:26:05.210038 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:26:05.211105 17504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:05.298261 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 17306
I20260809 05:26:05.324779 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 17515
I20260809 05:26:05.332238 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:05.325600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48630 (local address 127.22.1.67:38757)
0809 05:26:05.325799 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:05.325805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:05.325886 (+ 81us) server_negotiation.cc:407] Connection header received
0809 05:26:05.326094 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:05.326100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:05.326190 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:05.326304 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:05.326730 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.327596 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.328949 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:05.329474 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:05.329604 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:05.330720 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:05.330729 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:05.330747 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:05.331414 (+ 667us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:05.331485 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:05.331508 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:05.331605 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:05.331764 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 05:26:05.331883 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260809 05:26:05.332904 17687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:05.455300 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 17515
I20260809 05:26:05.478847 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 16961
I20260809 05:26:05.480087 17044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:05.587479 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 16961
2026-08-09T05:26:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2837 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:05 dist-test-slave-rmhg krb5kdc[17765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:05 dist-test-slave-rmhg krb5kdc[17765](info): set up 2 sockets
Aug 09 05:26:05 dist-test-slave-rmhg krb5kdc[17765](info): commencing operation
krb5kdc: starting...
W20260809 05:26:06.814128 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:07 dist-test-slave-rmhg krb5kdc[17765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:07.122599 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43499
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:43499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:07.470754 17786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:07.471077 17786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:07.471199 17786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:07.478559 17786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:07.478691 17786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:07.478750 17786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:07.478828 17786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:07.478879 17786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:07.488937 17786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:43499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:07.490386 17786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:07.491824 17786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:07.504062 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:07.504127 17801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:07.504102 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:07.505178 17786 server_base.cc:1034] running on GCE node
I20260809 05:26:07.506083 17786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:07.507628 17786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:07.508841 17786 hybrid_clock.cc:648] HybridClock initialized: now 1786253167508777 us; error 56 us; skew 500 ppm
I20260809 05:26:07.509236 17786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:07 dist-test-slave-rmhg krb5kdc[17765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:07.512250 17786 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:07.513837 17786 webserver.cc:460] Webserver started at http://127.22.1.126:34159/ using document root <none> and password file <none>
I20260809 05:26:07.514539 17786 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:07.514655 17786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:07.514954 17786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:07.518013 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "2b12e18b872b4a87b36fdf4bc0ea295f"
format_stamp: "Formatted at 2026-08-09 05:26:07 on dist-test-slave-rmhg"
server_key: "dc8e1437a09b73533e62e05dc8bce1f0"
server_key_iv: "b2eb2aef523339c35180666f36b3e794"
server_key_version: "encryptionkey@0"
I20260809 05:26:07.518752 17786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "2b12e18b872b4a87b36fdf4bc0ea295f"
format_stamp: "Formatted at 2026-08-09 05:26:07 on dist-test-slave-rmhg"
server_key: "dc8e1437a09b73533e62e05dc8bce1f0"
server_key_iv: "b2eb2aef523339c35180666f36b3e794"
server_key_version: "encryptionkey@0"
I20260809 05:26:07.524013 17786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:07.527657 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:07.529079 17786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260809 05:26:07.529282 17786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "2b12e18b872b4a87b36fdf4bc0ea295f"
format_stamp: "Formatted at 2026-08-09 05:26:07 on dist-test-slave-rmhg"
server_key: "dc8e1437a09b73533e62e05dc8bce1f0"
server_key_iv: "b2eb2aef523339c35180666f36b3e794"
server_key_version: "encryptionkey@0"
I20260809 05:26:07.529481 17786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:07.542774 17786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:07.547403 17786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:07.547706 17786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:07.565768 17786 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:43499
I20260809 05:26:07.565802 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:43499 every 8 connection(s)
I20260809 05:26:07.567212 17786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:07.571334 17892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:07.573817 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 17786
I20260809 05:26:07.574044 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:07.574597 22533 external_mini_cluster.cc:1428] Setting key f6a43e1d8ab159791448ca77e296cbda
I20260809 05:26:07.585062 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: Bootstrap starting.
I20260809 05:26:07.589254 17892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:07.590662 17892 log.cc:824] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:26:07 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:07.594120 17892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: No bootstrap required, opened a new log
I20260809 05:26:07.599862 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:07.576743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.22.1.126:43499)
0809 05:26:07.577717 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:07.577730 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:07.577785 (+ 55us) server_negotiation.cc:407] Connection header received
0809 05:26:07.579630 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:07.579661 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:07.580131 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:07.580508 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:07.582608 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:07.584002 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:07.586060 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:07.586785 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:07.586933 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:07.592322 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:07.592332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:07.592350 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:07.592386 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:07.595455 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:07.596228 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:07.596239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:07.596245 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:07.596426 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:07.596847 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:07.596851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:07.596853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:07.597492 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:07.597805 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:07.598309 (+ 504us) server_negotiation.cc:299] Negotiation successful
0809 05:26:07.598764 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":424,"threads_started":1}
I20260809 05:26:07.602376 17892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } }
I20260809 05:26:07.602867 17892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:07.602986 17892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b12e18b872b4a87b36fdf4bc0ea295f, State: Initialized, Role: FOLLOWER
I20260809 05:26:07.603538 17892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } }
I20260809 05:26:07.603766 17892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:07.603888 17892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:07.604067 17892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:07.606392 17892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } }
I20260809 05:26:07.606907 17892 leader_election.cc:302] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b12e18b872b4a87b36fdf4bc0ea295f; no voters:
I20260809 05:26:07.607532 17892 leader_election.cc:288] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:07.607622 17903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:07.608341 17903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [term 1 LEADER]: Becoming Leader. State: Replica: 2b12e18b872b4a87b36fdf4bc0ea295f, State: Running, Role: LEADER
I20260809 05:26:07.608865 17903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } }
I20260809 05:26:07.609701 17892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:07.611536 17904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } } }
I20260809 05:26:07.612078 17904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:07.614194 17906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b12e18b872b4a87b36fdf4bc0ea295f. Latest consensus state: current_term: 1 leader_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b12e18b872b4a87b36fdf4bc0ea295f" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43499 } } }
I20260809 05:26:07.614452 17911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:07.614555 17906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:07.620124 17911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:07.629859 17911 catalog_manager.cc:1347] Generated new cluster ID: ed7a3827d2a14eafaac5b89a546263ce
I20260809 05:26:07.629990 17911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:07.649951 17911 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:07.651889 17911 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:07.659821 17911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: Generated new TSK 0
I20260809 05:26:07.661087 17911 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:26:07.732717 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:08.087033 17933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:08.087349 17933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:08.087484 17933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:08.094830 17933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:08.094964 17933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:08.095118 17933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:08.106525 17933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:08.107894 17933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:08.109314 17933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:08.122855 17942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:08.122887 17944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:08.123064 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:08.124042 17933 server_base.cc:1034] running on GCE node
I20260809 05:26:08.124734 17933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:08.125823 17933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:08.127053 17933 hybrid_clock.cc:648] HybridClock initialized: now 1786253168126997 us; error 40 us; skew 500 ppm
I20260809 05:26:08.127393 17933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:08 dist-test-slave-rmhg krb5kdc[17765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:08.130671 17933 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:26:08.132200 17933 webserver.cc:460] Webserver started at http://127.22.1.65:36181/ using document root <none> and password file <none>
I20260809 05:26:08.132925 17933 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:08.133097 17933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:08.133484 17933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:08.136777 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "615be303a03c48ec8704da21024325ea"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "dcf08fa3fd60dd27534317fae55c2e9c"
server_key_iv: "3344d22b8d758e94ea096f3ad4e7c638"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.137727 17933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "615be303a03c48ec8704da21024325ea"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "dcf08fa3fd60dd27534317fae55c2e9c"
server_key_iv: "3344d22b8d758e94ea096f3ad4e7c638"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.143726 17933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260809 05:26:08.147857 17955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.149389 17933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260809 05:26:08.149569 17933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "615be303a03c48ec8704da21024325ea"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "dcf08fa3fd60dd27534317fae55c2e9c"
server_key_iv: "3344d22b8d758e94ea096f3ad4e7c638"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.149770 17933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:08.163546 17933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:08.167914 17933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:08.168213 17933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:08.169270 17933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:08.170696 17933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:08.170770 17933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.170894 17933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:08.170946 17933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.202612 17933 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:40195
I20260809 05:26:08.202651 18123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:40195 every 8 connection(s)
I20260809 05:26:08.204001 17933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:08.206225 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 17933
I20260809 05:26:08.206393 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:08.206920 22533 external_mini_cluster.cc:1428] Setting key f6daa589d74af70d79693dd0cf7604b6
Aug 09 05:26:08 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:08.226280 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:08.206770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:59115 (local address 127.22.1.126:43499)
0809 05:26:08.207359 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:08.207365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:08.208647 (+ 1282us) server_negotiation.cc:407] Connection header received
0809 05:26:08.210790 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:08.210796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:08.210884 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:08.211009 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:08.212725 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.213535 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.215242 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.215669 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.215789 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:08.219752 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:08.219759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:08.219762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:08.219804 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:08.222646 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:08.223679 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:08.223683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:08.223685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:08.223816 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:08.224325 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:08.224328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:08.224330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:08.224643 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:08.224756 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:08.225785 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0809 05:26:08.226010 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":320,"threads_started":1}
I20260809 05:26:08.227758 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:08.207389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43499 (local address 127.22.1.65:59115)
0809 05:26:08.208483 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:08.208529 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:08.210500 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:08.211279 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:08.211289 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:08.211875 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:08.212533 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:08.212552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.213704 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.213708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:08.215086 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:08.215102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.215930 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.215937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:08.216208 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:08.216921 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:08.216947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:08.219451 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:08.222860 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:08.222872 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:08.222893 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:08.223557 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:08.223938 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:08.223943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:08.223947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:08.224229 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:08.224771 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:08.224779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:08.225343 (+ 564us) client_negotiation.cc:769] Sending connection context
0809 05:26:08.225658 (+ 315us) client_negotiation.cc:240] Negotiation successful
0809 05:26:08.226299 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":611,"threads_started":1}
I20260809 05:26:08.229454 18124 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43499
I20260809 05:26:08.229780 18124 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:08.230490 18124 heartbeater.cc:511] Master 127.22.1.126:43499 requested a full tablet report, sending...
I20260809 05:26:08.232883 17837 ts_manager.cc:194] Registered new tserver with Master: 615be303a03c48ec8704da21024325ea (127.22.1.65:40195)
I20260809 05:26:08.234959 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:59115
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:26:08.274660 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:08.635788 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:08.636070 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:08.636193 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:08.643400 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:08.643523 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:08.643692 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:08.653636 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:08.655066 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:08.656432 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:08.670022 18138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:08.670054 18137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:08.669997 18141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:08.671041 18131 server_base.cc:1034] running on GCE node
I20260809 05:26:08.671663 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:08.672636 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:08.673853 18131 hybrid_clock.cc:648] HybridClock initialized: now 1786253168673794 us; error 45 us; skew 500 ppm
I20260809 05:26:08.674206 18131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:08 dist-test-slave-rmhg krb5kdc[17765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:08.677423 18131 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:26:08.678998 18131 webserver.cc:460] Webserver started at http://127.22.1.66:42607/ using document root <none> and password file <none>
I20260809 05:26:08.679709 18131 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:08.679826 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:08.680150 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:08.683208 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "35d187aa976b4074b4694aec6151ffb0"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "2fd2ca1dfacd379de144e7ec4c6eb6b1"
server_key_iv: "4f9d23294ba66f1e4f3fbb75e0b4bb2a"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.683964 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "35d187aa976b4074b4694aec6151ffb0"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "2fd2ca1dfacd379de144e7ec4c6eb6b1"
server_key_iv: "4f9d23294ba66f1e4f3fbb75e0b4bb2a"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.689357 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 05:26:08.693101 18154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.694617 18131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:08.694795 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "35d187aa976b4074b4694aec6151ffb0"
format_stamp: "Formatted at 2026-08-09 05:26:08 on dist-test-slave-rmhg"
server_key: "2fd2ca1dfacd379de144e7ec4c6eb6b1"
server_key_iv: "4f9d23294ba66f1e4f3fbb75e0b4bb2a"
server_key_version: "encryptionkey@0"
I20260809 05:26:08.695000 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:08.712497 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:08.717365 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:08.717660 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:08.718662 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:08.720043 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:08.720124 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.720250 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:08.720304 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:08.751657 18131 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:45751
I20260809 05:26:08.751708 18307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:45751 every 8 connection(s)
I20260809 05:26:08.752975 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:08.756407 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 18131
I20260809 05:26:08.756580 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:08.757030 22533 external_mini_cluster.cc:1428] Setting key 05f8e037d0e71db7cb6ecdc666449c9b
Aug 09 05:26:08 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253168, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:08.773615 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:08.755756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:49119 (local address 127.22.1.126:43499)
0809 05:26:08.756197 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:08.756202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:08.757335 (+ 1133us) server_negotiation.cc:407] Connection header received
0809 05:26:08.759178 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:08.759184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:08.759261 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:08.759381 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:08.760876 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.761869 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.763402 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.763936 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.764061 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:08.767583 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:08.767590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:08.767594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:08.767633 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:08.770549 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:08.771244 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:08.771247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:08.771250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:08.771375 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:08.771838 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:08.771842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:08.771844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:08.772132 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:08.772241 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:08.773125 (+ 884us) server_negotiation.cc:299] Negotiation successful
0809 05:26:08.773342 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260809 05:26:08.775038 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:08.756312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43499 (local address 127.22.1.66:49119)
0809 05:26:08.757133 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:08.757212 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:08.758922 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:08.759587 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:08.759595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:08.760112 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:08.760693 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:08.760712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.762012 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.762015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:08.763261 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:08.763273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:08.764128 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:08.764132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:08.764336 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:08.764982 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:08.765004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:08.767333 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:08.770682 (+ 3349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:08.770689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:08.770703 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:08.771131 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:08.771470 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:08.771474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:08.771478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:08.771732 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:08.772251 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:08.772259 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:08.772735 (+ 476us) client_negotiation.cc:769] Sending connection context
0809 05:26:08.773031 (+ 296us) client_negotiation.cc:240] Negotiation successful
0809 05:26:08.773608 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":329,"threads_started":1}
I20260809 05:26:08.776664 18308 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43499
I20260809 05:26:08.777027 18308 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:08.778010 18308 heartbeater.cc:511] Master 127.22.1.126:43499 requested a full tablet report, sending...
I20260809 05:26:08.779922 17837 ts_manager.cc:194] Registered new tserver with Master: 35d187aa976b4074b4694aec6151ffb0 (127.22.1.66:45751)
I20260809 05:26:08.780826 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:49119
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:26:08.822542 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:09.164592 18316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:09.164897 18316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:09.165040 18316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:09.172245 18316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:09.172361 18316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:09.172531 18316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:09.182667 18316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:43499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:09.184139 18316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:09.185559 18316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:09.198410 18323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:09.198493 18325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:09.198431 18322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:09.199358 18316 server_base.cc:1034] running on GCE node
I20260809 05:26:09.199999 18316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:09.200927 18316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:09.202180 18316 hybrid_clock.cc:648] HybridClock initialized: now 1786253169202105 us; error 55 us; skew 500 ppm
I20260809 05:26:09.202570 18316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:09 dist-test-slave-rmhg krb5kdc[17765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253169, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:09.206593 18316 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:26:09.208423 18316 webserver.cc:460] Webserver started at http://127.22.1.67:37125/ using document root <none> and password file <none>
I20260809 05:26:09.209199 18316 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:09.209327 18316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:09.209599 18316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:09.212637 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "c78bd33f314449ec902b227201575403"
format_stamp: "Formatted at 2026-08-09 05:26:09 on dist-test-slave-rmhg"
server_key: "213b66cb0965fcf101d4f9fbf118670e"
server_key_iv: "321ec0cb88a62ce317350ed344114b58"
server_key_version: "encryptionkey@0"
I20260809 05:26:09.213407 18316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "c78bd33f314449ec902b227201575403"
format_stamp: "Formatted at 2026-08-09 05:26:09 on dist-test-slave-rmhg"
server_key: "213b66cb0965fcf101d4f9fbf118670e"
server_key_iv: "321ec0cb88a62ce317350ed344114b58"
server_key_version: "encryptionkey@0"
I20260809 05:26:09.218779 18316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260809 05:26:09.222797 18334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:09.224324 18316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:09.224525 18316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "c78bd33f314449ec902b227201575403"
format_stamp: "Formatted at 2026-08-09 05:26:09 on dist-test-slave-rmhg"
server_key: "213b66cb0965fcf101d4f9fbf118670e"
server_key_iv: "321ec0cb88a62ce317350ed344114b58"
server_key_version: "encryptionkey@0"
I20260809 05:26:09.224747 18316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:09.236779 18316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:09.238121 18124 heartbeater.cc:503] Master 127.22.1.126:43499 was elected leader, sending a full tablet report...
I20260809 05:26:09.241071 18316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:09.241364 18316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:09.242308 18316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:09.243593 18316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:09.243667 18316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:09.243757 18316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:09.243800 18316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:09.272928 18316 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:32923
I20260809 05:26:09.272934 18514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:32923 every 8 connection(s)
I20260809 05:26:09.274348 18316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:09.283308 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 18316
I20260809 05:26:09.283481 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:09.283938 22533 external_mini_cluster.cc:1428] Setting key 0b114ce1234fd6db2bfed3d1db324d24
Aug 09 05:26:09 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253169, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:09.293623 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.277011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:44373 (local address 127.22.1.126:43499)
0809 05:26:09.277488 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.277492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.278509 (+ 1017us) server_negotiation.cc:407] Connection header received
0809 05:26:09.280348 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.280353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.280445 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.280539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:09.282099 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.282875 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.284511 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.284933 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.285052 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.288270 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:09.288274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:09.288278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:09.288309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:09.290705 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.291391 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.291395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.291398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.291509 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.291913 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.291917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.291919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.292184 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:09.292321 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.292996 (+ 675us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.293250 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260809 05:26:09.294557 18520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.277631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:43499 (local address 127.22.1.67:44373)
0809 05:26:09.278343 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.278384 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.280085 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.280786 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.280793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.281355 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:09.281929 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.281947 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.283028 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.283032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.284352 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.284370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.285175 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.285183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.285402 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.286012 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:09.286036 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:09.288077 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:09.290836 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:09.290845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:09.290861 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:09.291269 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:09.291611 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:09.291616 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:09.291620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:09.291800 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:09.292304 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:09.292312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:09.292639 (+ 327us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.292867 (+ 228us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.293411 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":271,"threads_started":1}
I20260809 05:26:09.296002 18516 heartbeater.cc:347] Connected to a master server at 127.22.1.126:43499
I20260809 05:26:09.296283 18516 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:09.296967 18516 heartbeater.cc:511] Master 127.22.1.126:43499 requested a full tablet report, sending...
I20260809 05:26:09.298435 17837 ts_manager.cc:194] Registered new tserver with Master: c78bd33f314449ec902b227201575403 (127.22.1.67:32923)
I20260809 05:26:09.299407 17837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:44373
I20260809 05:26:09.311693 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:09.329037 18519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.315763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.22.1.126:43499)
0809 05:26:09.315940 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.315947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.316339 (+ 392us) server_negotiation.cc:407] Connection header received
0809 05:26:09.316528 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.316532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.316613 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.316695 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:09.318169 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.318917 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.320387 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.320789 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.320900 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.323663 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:09.323668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:09.323671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:09.323703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:09.326077 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.326834 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.326838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.326840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.326959 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.327418 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.327422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.327423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.327815 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:09.328073 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.328529 (+ 456us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.328744 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 05:26:09.332998 17837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:09.340214 17837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:09.373976 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.353479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.22.1.67:32923)
0809 05:26:09.365977 (+ 12498us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.365987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.366033 (+ 46us) server_negotiation.cc:407] Connection header received
0809 05:26:09.366127 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.366134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.366345 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.366531 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:09.367103 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.368688 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.370860 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.372306 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.372441 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.372694 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.373464 (+ 770us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.373652 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12305,"thread_start_us":12148,"threads_started":1}
I20260809 05:26:09.374037 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.355149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.22.1.65:40195)
0809 05:26:09.363733 (+ 8584us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.363741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.366441 (+ 2700us) server_negotiation.cc:407] Connection header received
0809 05:26:09.366649 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.366657 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.366876 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.367078 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:09.367644 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.369099 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.371536 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.373000 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.373126 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.373392 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.373584 (+ 192us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.373728 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8397,"thread_start_us":8418,"threads_started":1}
I20260809 05:26:09.374006 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.353374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:45751 (local address 127.0.0.1:34630)
0809 05:26:09.364304 (+ 10930us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.364326 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.364523 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.365455 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.365464 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.365512 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:09.366061 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.366081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.368189 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.368194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.370812 (+ 2618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.370828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.372848 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.372853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.373058 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.373137 (+ 79us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.373302 (+ 165us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.373506 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10767,"mutex_wait_us":144,"thread_start_us":10595,"threads_started":1}
I20260809 05:26:09.374644 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.353936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.22.1.66:45751)
0809 05:26:09.363645 (+ 9709us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.363656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.364679 (+ 1023us) server_negotiation.cc:407] Connection header received
0809 05:26:09.364822 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.364832 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.365132 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.366306 (+ 1174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:09.366555 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.368035 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.370979 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.372350 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.372458 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.372663 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.374133 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.374314 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9508,"thread_start_us":9350,"threads_started":1}
I20260809 05:26:09.374011 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.354482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40195 (local address 127.0.0.1:53362)
0809 05:26:09.366302 (+ 11820us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.366320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.366439 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.367100 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.367104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.367130 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:09.367448 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.367461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.369261 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.369266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.371338 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.371352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.373246 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.373252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.373413 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.373443 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.373534 (+ 91us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.373650 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11674,"mutex_wait_us":44,"thread_start_us":7710,"threads_started":1}
I20260809 05:26:09.374845 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.353297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:32923 (local address 127.0.0.1:50458)
0809 05:26:09.362740 (+ 9443us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.362816 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.363073 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.366550 (+ 3477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.366555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.366578 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:09.366920 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.366931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.368853 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.368858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.370689 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.370703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.372847 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.372855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.373127 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.373202 (+ 75us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.373301 (+ 99us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.374428 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9090,"mutex_wait_us":77,"thread_start_us":12724,"threads_started":1}
I20260809 05:26:09.378656 18222 tablet_service.cc:1467] Processing CreateTablet for tablet b9f349f75e63440badb771ca9f8c95cb (DEFAULT_TABLE table=test-table [id=9e2a7360de28416594700bfa95191575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:09.378669 18024 tablet_service.cc:1467] Processing CreateTablet for tablet b9f349f75e63440badb771ca9f8c95cb (DEFAULT_TABLE table=test-table [id=9e2a7360de28416594700bfa95191575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:09.380110 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f349f75e63440badb771ca9f8c95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:09.380208 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f349f75e63440badb771ca9f8c95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:09.381484 18409 tablet_service.cc:1467] Processing CreateTablet for tablet b9f349f75e63440badb771ca9f8c95cb (DEFAULT_TABLE table=test-table [id=9e2a7360de28416594700bfa95191575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:09.383119 18409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f349f75e63440badb771ca9f8c95cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:09.392341 18543 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Bootstrap starting.
I20260809 05:26:09.395334 18544 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Bootstrap starting.
I20260809 05:26:09.396950 18543 tablet_bootstrap.cc:654] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:09.398034 18545 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Bootstrap starting.
I20260809 05:26:09.398500 18543 log.cc:824] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:09.399137 18544 tablet_bootstrap.cc:654] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:09.400470 18544 log.cc:824] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:09.401592 18543 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: No bootstrap required, opened a new log
I20260809 05:26:09.401885 18543 ts_tablet_manager.cc:1397] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 05:26:09.401901 18545 tablet_bootstrap.cc:654] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:09.403092 18545 log.cc:824] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:09.403630 18544 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: No bootstrap required, opened a new log
I20260809 05:26:09.403896 18544 ts_tablet_manager.cc:1397] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 05:26:09.406971 18545 tablet_bootstrap.cc:492] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: No bootstrap required, opened a new log
I20260809 05:26:09.407236 18545 ts_tablet_manager.cc:1397] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260809 05:26:09.409759 18543 raft_consensus.cc:348] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.410133 18543 raft_consensus.cc:374] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:09.410236 18543 raft_consensus.cc:729] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d187aa976b4074b4694aec6151ffb0, State: Initialized, Role: FOLLOWER
I20260809 05:26:09.410823 18543 consensus_queue.cc:260] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.414029 18544 raft_consensus.cc:348] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.414470 18544 raft_consensus.cc:374] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:09.414577 18544 raft_consensus.cc:729] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615be303a03c48ec8704da21024325ea, State: Initialized, Role: FOLLOWER
I20260809 05:26:09.415180 18544 consensus_queue.cc:260] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.418115 18545 raft_consensus.cc:348] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.418643 18545 raft_consensus.cc:374] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:09.418757 18545 raft_consensus.cc:729] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78bd33f314449ec902b227201575403, State: Initialized, Role: FOLLOWER
I20260809 05:26:09.419523 18545 consensus_queue.cc:260] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.420828 18516 heartbeater.cc:503] Master 127.22.1.126:43499 was elected leader, sending a full tablet report...
I20260809 05:26:09.420938 18308 heartbeater.cc:503] Master 127.22.1.126:43499 was elected leader, sending a full tablet report...
I20260809 05:26:09.421840 18543 ts_tablet_manager.cc:1428] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Time spent starting tablet: real 0.020s user 0.012s sys 0.008s
I20260809 05:26:09.423455 18544 ts_tablet_manager.cc:1428] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Time spent starting tablet: real 0.019s user 0.008s sys 0.011s
I20260809 05:26:09.424850 18545 ts_tablet_manager.cc:1428] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
W20260809 05:26:09.456487 18125 tablet.cc:2369] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:09.505131 18309 tablet.cc:2369] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:09.525945 18517 tablet.cc:2369] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:26:09.566537 18551 raft_consensus.cc:482] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:09.567021 18551 raft_consensus.cc:504] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.568879 18551 leader_election.cc:288] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c78bd33f314449ec902b227201575403 (127.22.1.67:32923), 615be303a03c48ec8704da21024325ea (127.22.1.65:40195)
I20260809 05:26:09.577546 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.569512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:43105 (local address 127.22.1.65:40195)
0809 05:26:09.569847 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.569854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.571167 (+ 1313us) server_negotiation.cc:407] Connection header received
0809 05:26:09.571333 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.571338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.571413 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.571510 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:09.572144 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.573282 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.575660 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.576461 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.576588 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.576698 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.577120 (+ 422us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.577292 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260809 05:26:09.577715 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.569941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:52961 (local address 127.22.1.67:32923)
0809 05:26:09.570168 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.570175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.570887 (+ 712us) server_negotiation.cc:407] Connection header received
0809 05:26:09.571177 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.571183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.571256 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.571352 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:09.572019 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.572821 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.575482 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.576725 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.576858 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.576964 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.577320 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.577467 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 05:26:09.578120 18559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.569392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:40195 (local address 127.22.1.66:43105)
0809 05:26:09.571048 (+ 1656us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.571065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.571199 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.571550 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.571554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.571583 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:09.571968 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.571982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.573431 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.573446 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.575511 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.575522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.576697 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.576703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.576880 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.576915 (+ 35us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.577061 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.577244 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":23,"thread_start_us":1305,"threads_started":1}
I20260809 05:26:09.578442 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.569300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:32923 (local address 127.22.1.66:52961)
0809 05:26:09.570734 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:09.570757 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:09.571010 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:09.571379 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:09.571383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:09.571407 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:09.571837 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.571855 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.572961 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.572965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.575288 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:09.575305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.576946 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.576952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:09.577109 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.577139 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 05:26:09.577866 (+ 727us) client_negotiation.cc:240] Negotiation successful
0809 05:26:09.578028 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":74,"thread_start_us":1032,"threads_started":1}
I20260809 05:26:09.579300 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f349f75e63440badb771ca9f8c95cb" candidate_uuid: "35d187aa976b4074b4694aec6151ffb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615be303a03c48ec8704da21024325ea" is_pre_election: true
I20260809 05:26:09.579632 18442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f349f75e63440badb771ca9f8c95cb" candidate_uuid: "35d187aa976b4074b4694aec6151ffb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd33f314449ec902b227201575403" is_pre_election: true
I20260809 05:26:09.580010 18055 raft_consensus.cc:2454] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d187aa976b4074b4694aec6151ffb0 in term 0.
I20260809 05:26:09.580178 18442 raft_consensus.cc:2454] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35d187aa976b4074b4694aec6151ffb0 in term 0.
I20260809 05:26:09.580838 18157 leader_election.cc:302] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d187aa976b4074b4694aec6151ffb0, 615be303a03c48ec8704da21024325ea; no voters:
I20260809 05:26:09.581307 18551 raft_consensus.cc:2781] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:09.581498 18551 raft_consensus.cc:482] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:09.581611 18551 raft_consensus.cc:3037] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:09.583855 18551 raft_consensus.cc:504] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.584527 18551 leader_election.cc:288] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [CANDIDATE]: Term 1 election: Requested vote from peers c78bd33f314449ec902b227201575403 (127.22.1.67:32923), 615be303a03c48ec8704da21024325ea (127.22.1.65:40195)
I20260809 05:26:09.585031 18442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f349f75e63440badb771ca9f8c95cb" candidate_uuid: "35d187aa976b4074b4694aec6151ffb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd33f314449ec902b227201575403"
I20260809 05:26:09.585098 18055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f349f75e63440badb771ca9f8c95cb" candidate_uuid: "35d187aa976b4074b4694aec6151ffb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615be303a03c48ec8704da21024325ea"
I20260809 05:26:09.585263 18442 raft_consensus.cc:3037] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:09.585343 18055 raft_consensus.cc:3037] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:09.587513 18442 raft_consensus.cc:2454] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d187aa976b4074b4694aec6151ffb0 in term 1.
I20260809 05:26:09.587559 18055 raft_consensus.cc:2454] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d187aa976b4074b4694aec6151ffb0 in term 1.
I20260809 05:26:09.587977 18155 leader_election.cc:302] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d187aa976b4074b4694aec6151ffb0, c78bd33f314449ec902b227201575403; no voters:
I20260809 05:26:09.588310 18551 raft_consensus.cc:2781] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:09.588841 18551 raft_consensus.cc:686] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 1 LEADER]: Becoming Leader. State: Replica: 35d187aa976b4074b4694aec6151ffb0, State: Running, Role: LEADER
I20260809 05:26:09.589377 18551 consensus_queue.cc:237] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } }
I20260809 05:26:09.594281 17837 catalog_manager.cc:5495] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35d187aa976b4074b4694aec6151ffb0 (127.22.1.66). New cstate: current_term: 1 leader_uuid: "35d187aa976b4074b4694aec6151ffb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35d187aa976b4074b4694aec6151ffb0" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 45751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 } health_report { overall_health: UNKNOWN } } }
I20260809 05:26:09.616190 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.610821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.22.1.66:45751)
0809 05:26:09.611061 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.611066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.611086 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:26:09.611285 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.611289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.611371 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.611497 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:09.611814 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.612529 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.613913 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.614400 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.614510 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.615026 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:09.615608 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:09.615741 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.615852 (+ 111us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.615893 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:09.625257 18055 raft_consensus.cc:1261] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 1 FOLLOWER]: Refusing update from remote peer 35d187aa976b4074b4694aec6151ffb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:09.625290 18442 raft_consensus.cc:1261] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 1 FOLLOWER]: Refusing update from remote peer 35d187aa976b4074b4694aec6151ffb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:09.626282 18560 consensus_queue.cc:1035] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c78bd33f314449ec902b227201575403" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 32923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:09.626621 18551 consensus_queue.cc:1035] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "615be303a03c48ec8704da21024325ea" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:09.636806 18565 mvcc.cc:205] Tried to move back new op lower bound from 7316492982774038528 to 7316492982641975296. Current Snapshot: MvccSnapshot[applied={T|T < 7316492982774038528}]
I20260809 05:26:09.655750 17837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:09.656069 17837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:09.660344 17837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b12e18b872b4a87b36fdf4bc0ea295f: Sending DeleteTablet for 3 replicas of tablet b9f349f75e63440badb771ca9f8c95cb
I20260809 05:26:09.661820 18024 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f349f75e63440badb771ca9f8c95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:53362
I20260809 05:26:09.661816 18222 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f349f75e63440badb771ca9f8c95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:34630
I20260809 05:26:09.661820 18409 tablet_service.cc:1514] Processing DeleteTablet for tablet b9f349f75e63440badb771ca9f8c95cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:09 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:50458
I20260809 05:26:09.669180 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 17933
I20260809 05:26:09.672688 18582 tablet_replica.cc:330] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: stopping tablet replica
I20260809 05:26:09.673008 18584 tablet_replica.cc:330] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: stopping tablet replica
I20260809 05:26:09.673252 18582 raft_consensus.cc:2229] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:09.673447 18584 raft_consensus.cc:2229] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:26:09.673834 18582 raft_consensus.cc:2258] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:09.674127 18584 raft_consensus.cc:2258] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:09.674880 18581 tablet_replica.cc:330] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: stopping tablet replica
I20260809 05:26:09.675304 18581 raft_consensus.cc:2229] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:09.675817 18581 raft_consensus.cc:2258] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:09.676728 18584 ts_tablet_manager.cc:1905] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:09.678308 18581 ts_tablet_manager.cc:1905] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:09.678930 18582 ts_tablet_manager.cc:1905] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:09.684223 18584 ts_tablet_manager.cc:1918] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:09.684410 18584 log.cc:1196] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/b9f349f75e63440badb771ca9f8c95cb
I20260809 05:26:09.685011 18584 ts_tablet_manager.cc:1939] T b9f349f75e63440badb771ca9f8c95cb P 35d187aa976b4074b4694aec6151ffb0: Deleting consensus metadata
I20260809 05:26:09.687048 17815 catalog_manager.cc:4841] TS 35d187aa976b4074b4694aec6151ffb0 (127.22.1.66:45751): tablet b9f349f75e63440badb771ca9f8c95cb (table test-table [id=9e2a7360de28416594700bfa95191575]) successfully deleted
I20260809 05:26:09.691295 18582 ts_tablet_manager.cc:1918] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:09.691301 18581 ts_tablet_manager.cc:1918] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:09.691493 18581 log.cc:1196] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/b9f349f75e63440badb771ca9f8c95cb
I20260809 05:26:09.691681 18582 log.cc:1196] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/b9f349f75e63440badb771ca9f8c95cb
I20260809 05:26:09.692106 18581 ts_tablet_manager.cc:1939] T b9f349f75e63440badb771ca9f8c95cb P 615be303a03c48ec8704da21024325ea: Deleting consensus metadata
I20260809 05:26:09.692306 18582 ts_tablet_manager.cc:1939] T b9f349f75e63440badb771ca9f8c95cb P c78bd33f314449ec902b227201575403: Deleting consensus metadata
I20260809 05:26:09.694199 17814 catalog_manager.cc:4841] TS 615be303a03c48ec8704da21024325ea (127.22.1.65:40195): tablet b9f349f75e63440badb771ca9f8c95cb (table test-table [id=9e2a7360de28416594700bfa95191575]) successfully deleted
I20260809 05:26:09.694494 17811 catalog_manager.cc:4841] TS c78bd33f314449ec902b227201575403 (127.22.1.67:32923): tablet b9f349f75e63440badb771ca9f8c95cb (table test-table [id=9e2a7360de28416594700bfa95191575]) successfully deleted
Aug 09 05:26:09 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:26:09.707907 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.673498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.22.1.65:40195)
0809 05:26:09.673704 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.673711 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.688067 (+ 14356us) server_negotiation.cc:407] Connection header received
0809 05:26:09.688383 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.688389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.688480 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.688573 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:09.690456 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.691660 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.695301 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.695959 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.696099 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.700647 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:09.700668 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:09.700693 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:09.700748 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:09.704716 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.705600 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.705612 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.705615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.705789 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.706184 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.706190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.706193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.706706 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:09.707032 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.707267 (+ 235us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.707507 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:26:09.708668 18117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:09.814896 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 17933
I20260809 05:26:09.853015 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 18131
Aug 09 05:26:09 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:26:09.869884 18538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:09.853960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.22.1.66:45751)
0809 05:26:09.854193 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:09.854203 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:09.854224 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:09.854368 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:09.854373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:09.854436 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:09.854526 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:09.855824 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.856715 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.858048 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:09.858514 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:09.858642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:09.863343 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:09.863358 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:09.863377 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:09.863429 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:09.866823 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.867812 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.867825 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.867827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.867978 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:09.868381 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:09.868384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:09.868386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:09.868783 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:09.869032 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:09.869420 (+ 388us) server_negotiation.cc:299] Negotiation successful
0809 05:26:09.869609 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 05:26:09.870786 18301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:09.987020 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 18131
I20260809 05:26:10.012908 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 18316
Aug 09 05:26:10 dist-test-slave-rmhg krb5kdc[17765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:26:10.028910 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:10.013742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.22.1.67:32923)
0809 05:26:10.013920 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:10.013924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:10.013943 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:26:10.014070 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:10.014074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:10.014167 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:10.014251 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:10.015705 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:10.016516 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:10.018010 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:10.018414 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:10.018509 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:10.023302 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:10.023321 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:10.023339 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:10.023386 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:10.026112 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:10.026882 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:10.026889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:10.026891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:10.027065 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:10.027417 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:10.027420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:10.027421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:10.027784 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:10.028052 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:10.028314 (+ 262us) server_negotiation.cc:299] Negotiation successful
0809 05:26:10.028489 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:26:10.029543 18510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:10.118788 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 18316
I20260809 05:26:10.139760 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 17786
I20260809 05:26:10.141050 17885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:10.251597 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 17786
2026-08-09T05:26:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-09T05:26:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:10Z Disabled control of system clock
I20260809 05:26:10.343151 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:34877
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:10.697089 18601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:10.697425 18601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:10.697525 18601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:10.704850 18601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:10.704965 18601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:10.705003 18601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:10.705051 18601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:10.705094 18601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:10.714653 18601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:34877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:10.716015 18601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:10.717887 18601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:10.729578 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:10.729547 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:10.729686 18615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:10.730403 18601 server_base.cc:1034] running on GCE node
I20260809 05:26:10.731168 18601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:10.732599 18601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:10.733801 18601 hybrid_clock.cc:648] HybridClock initialized: now 1786253170733746 us; error 50 us; skew 500 ppm
I20260809 05:26:10.734149 18601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:10.736420 18601 webserver.cc:460] Webserver started at http://127.22.1.126:43811/ using document root <none> and password file <none>
I20260809 05:26:10.737138 18601 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:10.737283 18601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:10.737610 18601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:10.740495 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "3ba5d9d50b8149aaa82078cfb8889685"
format_stamp: "Formatted at 2026-08-09 05:26:10 on dist-test-slave-rmhg"
server_key: "68fbf0f4b6251d067a7c2fef9fa09768"
server_key_iv: "088bb91b222645c0db22e4366eeafddd"
server_key_version: "encryptionkey@0"
I20260809 05:26:10.741322 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "3ba5d9d50b8149aaa82078cfb8889685"
format_stamp: "Formatted at 2026-08-09 05:26:10 on dist-test-slave-rmhg"
server_key: "68fbf0f4b6251d067a7c2fef9fa09768"
server_key_iv: "088bb91b222645c0db22e4366eeafddd"
server_key_version: "encryptionkey@0"
I20260809 05:26:10.746356 18601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260809 05:26:10.749886 18623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:10.751266 18601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:10.751438 18601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "3ba5d9d50b8149aaa82078cfb8889685"
format_stamp: "Formatted at 2026-08-09 05:26:10 on dist-test-slave-rmhg"
server_key: "68fbf0f4b6251d067a7c2fef9fa09768"
server_key_iv: "088bb91b222645c0db22e4366eeafddd"
server_key_version: "encryptionkey@0"
I20260809 05:26:10.751662 18601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:10.764317 18601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:10.765244 18601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:10.765527 18601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:10.782831 18601 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:34877
I20260809 05:26:10.782837 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:34877 every 8 connection(s)
I20260809 05:26:10.784278 18601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:10.788142 18700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:10.793861 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 18601
I20260809 05:26:10.794081 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:10.794553 22533 external_mini_cluster.cc:1428] Setting key 42d1dade9c0f372c505605c5b58abd42
I20260809 05:26:10.799547 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: Bootstrap starting.
I20260809 05:26:10.803027 18700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:10.804316 18700 log.cc:824] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:10.807709 18700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: No bootstrap required, opened a new log
I20260809 05:26:10.809330 18703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:10.796797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.22.1.126:34877)
0809 05:26:10.797935 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:10.797949 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:10.797994 (+ 45us) server_negotiation.cc:407] Connection header received
0809 05:26:10.799669 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:10.799697 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:10.800193 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:10.800534 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:10.801145 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:10.802516 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:10.804400 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:10.805493 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:10.805699 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:10.806867 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:10.806875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:10.806890 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:10.807442 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:10.807528 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:10.807551 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:10.807639 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:10.807926 (+ 287us) server_negotiation.cc:299] Negotiation successful
0809 05:26:10.808271 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":413,"threads_started":1}
I20260809 05:26:10.815493 18700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } }
I20260809 05:26:10.815858 18700 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:10.815955 18700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba5d9d50b8149aaa82078cfb8889685, State: Initialized, Role: FOLLOWER
I20260809 05:26:10.816534 18700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } }
I20260809 05:26:10.816751 18700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:10.816874 18700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:10.817031 18700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:10.819283 18700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } }
I20260809 05:26:10.819783 18700 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba5d9d50b8149aaa82078cfb8889685; no voters:
I20260809 05:26:10.820353 18700 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:10.820498 18706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:10.821219 18706 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba5d9d50b8149aaa82078cfb8889685, State: Running, Role: LEADER
I20260809 05:26:10.821748 18706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } }
I20260809 05:26:10.822489 18700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:10.824728 18709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba5d9d50b8149aaa82078cfb8889685. Latest consensus state: current_term: 1 leader_uuid: "3ba5d9d50b8149aaa82078cfb8889685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } } }
I20260809 05:26:10.824705 18707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba5d9d50b8149aaa82078cfb8889685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba5d9d50b8149aaa82078cfb8889685" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 34877 } } }
I20260809 05:26:10.825210 18709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:10.825366 18707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:10.826700 18714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:10.832566 18714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:10.842959 18714 catalog_manager.cc:1347] Generated new cluster ID: f134dfe2e6c849f3aa0a0f19bfc01a97
I20260809 05:26:10.843102 18714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:10.859817 18714 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:10.861330 18714 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:10.868793 18714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: Generated new TSK 0
I20260809 05:26:10.869721 18714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:10.877615 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--builtin_ntp_servers=127.22.1.84:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:11.227761 18735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:11.228096 18735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:11.228236 18735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:11.236065 18735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:11.236207 18735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:11.236369 18735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:11.245707 18735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:11.247087 18735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:11.248899 18735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:11.266096 18742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:11.266125 18743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:11.266163 18745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:11.267266 18735 server_base.cc:1034] running on GCE node
I20260809 05:26:11.267920 18735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:11.269078 18735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:11.270330 18735 hybrid_clock.cc:648] HybridClock initialized: now 1786253171270270 us; error 48 us; skew 500 ppm
I20260809 05:26:11.270725 18735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:11.273339 18735 webserver.cc:460] Webserver started at http://127.22.1.65:36729/ using document root <none> and password file <none>
I20260809 05:26:11.274134 18735 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:11.274261 18735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:11.274606 18735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:11.277820 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "e023c145880c404c9c3798890b49c577"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a52a83a93b8cf2b74e3f9c845a5c82be"
server_key_iv: "e322f0b969630bb847e95ef2e1335ac4"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.278684 18735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "e023c145880c404c9c3798890b49c577"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a52a83a93b8cf2b74e3f9c845a5c82be"
server_key_iv: "e322f0b969630bb847e95ef2e1335ac4"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.284231 18735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:26:11.287884 18752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.289376 18735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:11.289569 18735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "e023c145880c404c9c3798890b49c577"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a52a83a93b8cf2b74e3f9c845a5c82be"
server_key_iv: "e322f0b969630bb847e95ef2e1335ac4"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.289790 18735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:11.313517 18735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:11.314541 18735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:11.314846 18735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:11.316010 18735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:11.317808 18735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:11.317921 18735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.318035 18735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:11.318106 18735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.350256 18735 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:43293
I20260809 05:26:11.350338 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:43293 every 8 connection(s)
I20260809 05:26:11.351640 18735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:11.360091 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 18735
I20260809 05:26:11.360249 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:11.360694 22533 external_mini_cluster.cc:1428] Setting key 8f00a98311a6d89d6415b6ae7076a894
I20260809 05:26:11.363425 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--builtin_ntp_servers=127.22.1.84:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:11.365032 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:11.354530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:47583 (local address 127.22.1.126:34877)
0809 05:26:11.355001 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:11.355005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:11.356273 (+ 1268us) server_negotiation.cc:407] Connection header received
0809 05:26:11.358246 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:11.358252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:11.358342 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:11.358452 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:11.359591 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.360423 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.362187 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.362687 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.362800 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:11.364198 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:11.364204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:11.364207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:11.364224 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:11.364267 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:11.364279 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:11.364354 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:11.364585 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 05:26:11.364685 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260809 05:26:11.366093 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:11.355014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34877 (local address 127.22.1.65:47583)
0809 05:26:11.356088 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:11.356128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:11.357957 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:11.358700 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:11.358709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:11.358810 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:11.359418 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:11.359434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.360589 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.360593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:11.362049 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:11.362062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.362915 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.362923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:11.363162 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:11.363880 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:11.363883 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:11.363925 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:11.364019 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:11.364368 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:11.364379 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:11.364388 (+ 9us) client_negotiation.cc:769] Sending connection context
0809 05:26:11.364562 (+ 174us) client_negotiation.cc:240] Negotiation successful
0809 05:26:11.364989 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":304,"threads_started":1}
I20260809 05:26:11.367548 18921 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34877
I20260809 05:26:11.367892 18921 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:11.368830 18921 heartbeater.cc:511] Master 127.22.1.126:34877 requested a full tablet report, sending...
I20260809 05:26:11.371124 18650 ts_manager.cc:194] Registered new tserver with Master: e023c145880c404c9c3798890b49c577 (127.22.1.65:43293)
I20260809 05:26:11.372610 18650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.65:47583
W20260809 05:26:11.720011 18929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:11.720342 18929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:11.720453 18929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:11.728451 18929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:11.728574 18929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:11.728715 18929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:11.738459 18929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:11.739825 18929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:11.741731 18929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:11.755548 18938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:11.755501 18937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:11.755501 18943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:11.756278 18929 server_base.cc:1034] running on GCE node
I20260809 05:26:11.757025 18929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:11.758092 18929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:11.759315 18929 hybrid_clock.cc:648] HybridClock initialized: now 1786253171759259 us; error 47 us; skew 500 ppm
I20260809 05:26:11.759668 18929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:11.762152 18929 webserver.cc:460] Webserver started at http://127.22.1.66:42511/ using document root <none> and password file <none>
I20260809 05:26:11.762887 18929 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:11.762989 18929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:11.763312 18929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:11.766489 18929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a3a9fd61dce9e62969c2e99cdb5088f5"
server_key_iv: "be028f0c145931a9a49e79abf8caf817"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.767292 18929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a3a9fd61dce9e62969c2e99cdb5088f5"
server_key_iv: "be028f0c145931a9a49e79abf8caf817"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.772802 18929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:11.776559 18950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.778049 18929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:11.778220 18929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e"
format_stamp: "Formatted at 2026-08-09 05:26:11 on dist-test-slave-rmhg"
server_key: "a3a9fd61dce9e62969c2e99cdb5088f5"
server_key_iv: "be028f0c145931a9a49e79abf8caf817"
server_key_version: "encryptionkey@0"
I20260809 05:26:11.778430 18929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:11.819707 18929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:11.820602 18929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:11.820875 18929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:11.821972 18929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:11.823604 18929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:11.823706 18929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.823813 18929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:11.823885 18929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:11.855218 18929 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:41053
I20260809 05:26:11.855281 19115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:41053 every 8 connection(s)
I20260809 05:26:11.856594 18929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:11.866704 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 18929
I20260809 05:26:11.866890 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:11.867594 22533 external_mini_cluster.cc:1428] Setting key 8983d74bf6c3cc0343e8c3b6f17aa2df
I20260809 05:26:11.869721 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:11.859247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:40509 (local address 127.22.1.126:34877)
0809 05:26:11.859423 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:11.859428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:11.860853 (+ 1425us) server_negotiation.cc:407] Connection header received
0809 05:26:11.862716 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:11.862722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:11.862806 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:11.862897 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:11.864023 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.864774 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.866407 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.866845 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.866963 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:11.868716 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:11.868721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:11.868724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:11.868743 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:11.868788 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:11.868803 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:11.868937 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:11.869238 (+ 301us) server_negotiation.cc:299] Negotiation successful
0809 05:26:11.869353 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260809 05:26:11.870829 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:11.859774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34877 (local address 127.22.1.66:40509)
0809 05:26:11.860680 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:11.860725 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:11.862434 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:11.863164 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:11.863174 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:11.863273 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:11.863846 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:11.863862 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.864939 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.864943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:11.866256 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:11.866270 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:11.867205 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:11.867215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:11.867555 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:11.868353 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:11.868355 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:11.868400 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:11.868509 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:11.868946 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:11.868958 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:11.868968 (+ 10us) client_negotiation.cc:769] Sending connection context
0809 05:26:11.869215 (+ 247us) client_negotiation.cc:240] Negotiation successful
0809 05:26:11.869634 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":264,"threads_started":1}
I20260809 05:26:11.870904 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--builtin_ntp_servers=127.22.1.84:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:11.872421 19117 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34877
I20260809 05:26:11.872766 19117 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:11.873668 19117 heartbeater.cc:511] Master 127.22.1.126:34877 requested a full tablet report, sending...
I20260809 05:26:11.875293 18650 ts_manager.cc:194] Registered new tserver with Master: 59cd53c7b2aa4c7ab79b0de53c0a6a2e (127.22.1.66:41053)
I20260809 05:26:11.876323 18650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.66:40509
W20260809 05:26:12.210155 19120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:12.210428 19120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:12.210547 19120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:12.217607 19120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:12.217700 19120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:12.217839 19120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:12.227075 19120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:34877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:12.228417 19120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:12.230304 19120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:12.243527 19128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:12.243563 19133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:12.243602 19129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:12.244879 19120 server_base.cc:1034] running on GCE node
I20260809 05:26:12.245543 19120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:12.246532 19120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:12.247736 19120 hybrid_clock.cc:648] HybridClock initialized: now 1786253172247687 us; error 53 us; skew 500 ppm
I20260809 05:26:12.248070 19120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:12.250367 19120 webserver.cc:460] Webserver started at http://127.22.1.67:34887/ using document root <none> and password file <none>
I20260809 05:26:12.251078 19120 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:12.251185 19120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:12.251492 19120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:12.254403 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "813944b913c3410fb84c0a1e9aaf09e1"
format_stamp: "Formatted at 2026-08-09 05:26:12 on dist-test-slave-rmhg"
server_key: "fe98bdd2839bb02377ec48f6d693db7a"
server_key_iv: "db984a2dca2478965817fb0e93c84431"
server_key_version: "encryptionkey@0"
I20260809 05:26:12.255162 19120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "813944b913c3410fb84c0a1e9aaf09e1"
format_stamp: "Formatted at 2026-08-09 05:26:12 on dist-test-slave-rmhg"
server_key: "fe98bdd2839bb02377ec48f6d693db7a"
server_key_iv: "db984a2dca2478965817fb0e93c84431"
server_key_version: "encryptionkey@0"
I20260809 05:26:12.260157 19120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260809 05:26:12.263476 19143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:12.264760 19120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:26:12.264902 19120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "813944b913c3410fb84c0a1e9aaf09e1"
format_stamp: "Formatted at 2026-08-09 05:26:12 on dist-test-slave-rmhg"
server_key: "fe98bdd2839bb02377ec48f6d693db7a"
server_key_iv: "db984a2dca2478965817fb0e93c84431"
server_key_version: "encryptionkey@0"
I20260809 05:26:12.265092 19120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:12.288977 19120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:12.289876 19120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:12.290141 19120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:12.291155 19120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:12.292722 19120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:12.292817 19120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:12.292920 19120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:12.292992 19120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:12.323221 19120 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:39647
I20260809 05:26:12.323292 19294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:39647 every 8 connection(s)
I20260809 05:26:12.324537 19120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:12.330987 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 19120
I20260809 05:26:12.331140 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:12.331563 22533 external_mini_cluster.cc:1428] Setting key d4b297f8a9b19a095dc662dcfcb9f150
I20260809 05:26:12.337040 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.326982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:47921 (local address 127.22.1.126:34877)
0809 05:26:12.327213 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.327219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.328551 (+ 1332us) server_negotiation.cc:407] Connection header received
0809 05:26:12.330325 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.330329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.330399 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.330487 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:12.331647 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.332585 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.334725 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.335134 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.335228 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.336432 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:12.336436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:12.336439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:12.336457 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:12.336485 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:12.336496 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:12.336558 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.336724 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.336802 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:26:12.338021 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.327507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:34877 (local address 127.22.1.67:47921)
0809 05:26:12.328403 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.328440 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.330068 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.330747 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.330756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.330854 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:12.331478 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.331496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.332793 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.332799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.334604 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.334616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.335307 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.335312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.335505 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.336141 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:12.336144 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:12.336196 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:12.336280 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:12.336565 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:12.336571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:12.336576 (+ 5us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.336707 (+ 131us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.336992 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":254,"threads_started":1}
I20260809 05:26:12.339206 19298 heartbeater.cc:347] Connected to a master server at 127.22.1.126:34877
I20260809 05:26:12.339462 19298 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:12.340147 19298 heartbeater.cc:511] Master 127.22.1.126:34877 requested a full tablet report, sending...
I20260809 05:26:12.341475 18650 ts_manager.cc:194] Registered new tserver with Master: 813944b913c3410fb84c0a1e9aaf09e1 (127.22.1.67:39647)
I20260809 05:26:12.342231 18650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.67:47921
I20260809 05:26:12.346681 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:12.357527 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.350936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.22.1.126:34877)
0809 05:26:12.351153 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.351160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.351533 (+ 373us) server_negotiation.cc:407] Connection header received
0809 05:26:12.351687 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.351693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.351777 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.351886 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:12.352348 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.353457 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.354904 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.355574 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.355706 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.356856 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:12.356861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:12.356865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:12.356884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:12.356920 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:12.356932 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:12.357068 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.357226 (+ 158us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.357285 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260809 05:26:12.361461 18650 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:12.368300 18650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:12.376300 18921 heartbeater.cc:503] Master 127.22.1.126:34877 was elected leader, sending a full tablet report...
I20260809 05:26:12.398904 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.381283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:41053 (local address 127.0.0.1:58808)
0809 05:26:12.385617 (+ 4334us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.385665 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.385826 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.390447 (+ 4621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.390457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.390493 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:12.390989 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.391002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.393140 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.393145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.395790 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.395804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.397910 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.397916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.398142 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.398211 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.398333 (+ 122us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.398559 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4058,"thread_start_us":12095,"threads_started":1}
I20260809 05:26:12.400377 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.381865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.22.1.66:41053)
0809 05:26:12.389693 (+ 7828us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.389705 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.389743 (+ 38us) server_negotiation.cc:407] Connection header received
0809 05:26:12.389857 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.389865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.390175 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.390401 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:12.391323 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.393009 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.396049 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.397684 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.399481 (+ 1797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.399770 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.399845 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.400020 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7618,"thread_start_us":7483,"threads_started":1}
I20260809 05:26:12.403384 19315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.382220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:39647 (local address 127.0.0.1:55424)
0809 05:26:12.393175 (+ 10955us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.393196 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.393319 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.394288 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.394297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.394324 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:12.394754 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.394767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.398141 (+ 3374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.398146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.400805 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.400820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.402706 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.402710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.402833 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.402855 (+ 22us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.402929 (+ 74us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.403075 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10800,"thread_start_us":2984,"threads_started":1}
I20260809 05:26:12.403568 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.381718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.22.1.67:39647)
0809 05:26:12.392341 (+ 10623us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.392347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.393303 (+ 956us) server_negotiation.cc:407] Connection header received
0809 05:26:12.393455 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.393463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.393756 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.393987 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:12.394988 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.397963 (+ 2975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.401043 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.402274 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.402364 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.402559 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.403026 (+ 467us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.403207 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10388,"thread_start_us":10208,"threads_started":1}
I20260809 05:26:12.407433 19209 tablet_service.cc:1467] Processing CreateTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 (DEFAULT_TABLE table=test-table [id=36d0c3eab6054c07b2f0246ff3b0c350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:12.409094 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8514a2149e8a43f59e86ffcf1a379029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:12.412196 19022 tablet_service.cc:1467] Processing CreateTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 (DEFAULT_TABLE table=test-table [id=36d0c3eab6054c07b2f0246ff3b0c350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:12.413389 19022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8514a2149e8a43f59e86ffcf1a379029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:12.414486 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.393897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54566)
0809 05:26:12.399187 (+ 5290us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.399206 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.399306 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.407850 (+ 8544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.407855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.407885 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:12.408080 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.408092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.410074 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.410079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.411831 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.411846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.413714 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.413721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.413907 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.413944 (+ 37us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.414016 (+ 72us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.414169 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5166,"thread_start_us":14626,"threads_started":1}
I20260809 05:26:12.415607 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.394116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.22.1.65:43293)
0809 05:26:12.407167 (+ 13051us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.407177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.407214 (+ 37us) server_negotiation.cc:407] Connection header received
0809 05:26:12.407341 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.407351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.407621 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.407842 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:12.408287 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.409916 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.412014 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.413491 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.414729 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.415008 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.415086 (+ 78us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.415253 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12819,"thread_start_us":14039,"threads_started":1}
I20260809 05:26:12.421278 19325 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Bootstrap starting.
I20260809 05:26:12.424890 18832 tablet_service.cc:1467] Processing CreateTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 (DEFAULT_TABLE table=test-table [id=36d0c3eab6054c07b2f0246ff3b0c350]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:12.425796 19325 tablet_bootstrap.cc:654] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:12.425880 19326 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Bootstrap starting.
I20260809 05:26:12.426450 18832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8514a2149e8a43f59e86ffcf1a379029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:12.427181 19325 log.cc:824] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:12.429531 19326 tablet_bootstrap.cc:654] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:12.429852 19325 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: No bootstrap required, opened a new log
I20260809 05:26:12.430111 19325 ts_tablet_manager.cc:1397] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260809 05:26:12.430776 19326 log.cc:824] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:12.435422 19326 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: No bootstrap required, opened a new log
I20260809 05:26:12.435773 19326 ts_tablet_manager.cc:1397] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260809 05:26:12.438725 19325 raft_consensus.cc:348] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.439116 19325 raft_consensus.cc:374] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:12.439208 19325 raft_consensus.cc:729] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813944b913c3410fb84c0a1e9aaf09e1, State: Initialized, Role: FOLLOWER
I20260809 05:26:12.439813 19325 consensus_queue.cc:260] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.440877 19298 heartbeater.cc:503] Master 127.22.1.126:34877 was elected leader, sending a full tablet report...
I20260809 05:26:12.442060 19331 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Bootstrap starting.
I20260809 05:26:12.443706 19325 ts_tablet_manager.cc:1428] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260809 05:26:12.445704 19326 raft_consensus.cc:348] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.446131 19331 tablet_bootstrap.cc:654] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:12.446204 19326 raft_consensus.cc:374] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:12.446313 19326 raft_consensus.cc:729] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59cd53c7b2aa4c7ab79b0de53c0a6a2e, State: Initialized, Role: FOLLOWER
I20260809 05:26:12.447026 19326 consensus_queue.cc:260] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.447337 19331 log.cc:824] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:12.448413 19117 heartbeater.cc:503] Master 127.22.1.126:34877 was elected leader, sending a full tablet report...
I20260809 05:26:12.449173 19326 ts_tablet_manager.cc:1428] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260809 05:26:12.451614 19331 tablet_bootstrap.cc:492] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: No bootstrap required, opened a new log
I20260809 05:26:12.451864 19331 ts_tablet_manager.cc:1397] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260809 05:26:12.459158 19331 raft_consensus.cc:348] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.459645 19331 raft_consensus.cc:374] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:12.459761 19331 raft_consensus.cc:729] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e023c145880c404c9c3798890b49c577, State: Initialized, Role: FOLLOWER
I20260809 05:26:12.460325 19331 consensus_queue.cc:260] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.462044 19331 ts_tablet_manager.cc:1428] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
W20260809 05:26:12.576213 19299 tablet.cc:2369] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:12.604166 18922 tablet.cc:2369] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:12.608619 19118 tablet.cc:2369] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:26:12.763391 19335 raft_consensus.cc:482] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:12.763734 19335 raft_consensus.cc:504] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.765479 19335 leader_election.cc:288] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59cd53c7b2aa4c7ab79b0de53c0a6a2e (127.22.1.66:41053), 813944b913c3410fb84c0a1e9aaf09e1 (127.22.1.67:39647)
I20260809 05:26:12.774505 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.766051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:39657 (local address 127.22.1.66:41053)
0809 05:26:12.766240 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.766246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.767003 (+ 757us) server_negotiation.cc:407] Connection header received
0809 05:26:12.767193 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.767200 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.767293 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.767370 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:12.768235 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.769335 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.772312 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.773525 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.773663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.773758 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.774135 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.774257 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:26:12.774533 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.766063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:35429 (local address 127.22.1.67:39647)
0809 05:26:12.766240 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.766246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.766953 (+ 707us) server_negotiation.cc:407] Connection header received
0809 05:26:12.767194 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.767199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.767280 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.767370 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:12.768117 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.769317 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.772312 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.773547 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.773663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.773766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.774127 (+ 361us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.774253 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:12.774688 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.765902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:41053 (local address 127.22.1.65:39657)
0809 05:26:12.766878 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.766891 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.767038 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.767403 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.767408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.767442 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:12.768081 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.768094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.769488 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.769493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.772079 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.772089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.773750 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.773758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.773935 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.773997 (+ 62us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.774092 (+ 95us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.774274 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":249,"spinlock_wait_cycles":17920,"thread_start_us":561,"threads_started":1}
I20260809 05:26:12.775157 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.765767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:39647 (local address 127.22.1.65:35429)
0809 05:26:12.766817 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:12.766836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:12.767028 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:12.767403 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:12.767408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:12.767445 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:12.767962 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.767988 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.769490 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.769494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.772069 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:12.772086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.773720 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.773732 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:12.773929 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.773998 (+ 69us) client_negotiation.cc:769] Sending connection context
0809 05:26:12.774092 (+ 94us) client_negotiation.cc:240] Negotiation successful
0809 05:26:12.774274 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":313,"spinlock_wait_cycles":28416,"thread_start_us":724,"threads_started":1}
I20260809 05:26:12.775775 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8514a2149e8a43f59e86ffcf1a379029" candidate_uuid: "e023c145880c404c9c3798890b49c577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" is_pre_election: true
I20260809 05:26:12.776165 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8514a2149e8a43f59e86ffcf1a379029" candidate_uuid: "e023c145880c404c9c3798890b49c577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813944b913c3410fb84c0a1e9aaf09e1" is_pre_election: true
I20260809 05:26:12.776295 19051 raft_consensus.cc:2454] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e023c145880c404c9c3798890b49c577 in term 0.
I20260809 05:26:12.776670 19235 raft_consensus.cc:2454] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e023c145880c404c9c3798890b49c577 in term 0.
I20260809 05:26:12.776942 18755 leader_election.cc:302] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cd53c7b2aa4c7ab79b0de53c0a6a2e, e023c145880c404c9c3798890b49c577; no voters:
I20260809 05:26:12.777416 19335 raft_consensus.cc:2781] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:12.777580 19335 raft_consensus.cc:482] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:12.777681 19335 raft_consensus.cc:3037] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:12.779803 19335 raft_consensus.cc:504] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.780386 19335 leader_election.cc:288] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [CANDIDATE]: Term 1 election: Requested vote from peers 59cd53c7b2aa4c7ab79b0de53c0a6a2e (127.22.1.66:41053), 813944b913c3410fb84c0a1e9aaf09e1 (127.22.1.67:39647)
I20260809 05:26:12.780907 19235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8514a2149e8a43f59e86ffcf1a379029" candidate_uuid: "e023c145880c404c9c3798890b49c577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813944b913c3410fb84c0a1e9aaf09e1"
I20260809 05:26:12.780908 19051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8514a2149e8a43f59e86ffcf1a379029" candidate_uuid: "e023c145880c404c9c3798890b49c577" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e"
I20260809 05:26:12.781124 19051 raft_consensus.cc:3037] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:12.781124 19235 raft_consensus.cc:3037] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:12.784361 19235 raft_consensus.cc:2454] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e023c145880c404c9c3798890b49c577 in term 1.
I20260809 05:26:12.784364 19051 raft_consensus.cc:2454] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e023c145880c404c9c3798890b49c577 in term 1.
I20260809 05:26:12.784792 18755 leader_election.cc:302] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59cd53c7b2aa4c7ab79b0de53c0a6a2e, e023c145880c404c9c3798890b49c577; no voters:
I20260809 05:26:12.785104 19335 raft_consensus.cc:2781] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:12.789552 19335 raft_consensus.cc:686] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [term 1 LEADER]: Becoming Leader. State: Replica: e023c145880c404c9c3798890b49c577, State: Running, Role: LEADER
I20260809 05:26:12.790148 19335 consensus_queue.cc:237] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } }
I20260809 05:26:12.795770 18650 catalog_manager.cc:5495] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 reported cstate change: term changed from 0 to 1, leader changed from <none> to e023c145880c404c9c3798890b49c577 (127.22.1.65). New cstate: current_term: 1 leader_uuid: "e023c145880c404c9c3798890b49c577" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e023c145880c404c9c3798890b49c577" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 43293 } health_report { overall_health: HEALTHY } } }
I20260809 05:26:12.852226 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.846952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.22.1.65:43293)
0809 05:26:12.847179 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.847185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.847206 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:12.847415 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.847419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.847517 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.847699 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:12.848049 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.848734 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.850160 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.850571 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.850668 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.851157 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:12.851662 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:12.851786 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.851915 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.851955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 05:26:12.861274 19235 raft_consensus.cc:1261] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 1 FOLLOWER]: Refusing update from remote peer e023c145880c404c9c3798890b49c577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:12.861462 19051 raft_consensus.cc:1261] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 1 FOLLOWER]: Refusing update from remote peer e023c145880c404c9c3798890b49c577: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:12.862308 19338 consensus_queue.cc:1035] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59cd53c7b2aa4c7ab79b0de53c0a6a2e" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:12.862715 19335 consensus_queue.cc:1035] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577 [LEADER]: Connected to new peer: Peer: permanent_uuid: "813944b913c3410fb84c0a1e9aaf09e1" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:12.890743 18648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:12.891048 18648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:12.894949 18648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ba5d9d50b8149aaa82078cfb8889685: Sending DeleteTablet for 3 replicas of tablet 8514a2149e8a43f59e86ffcf1a379029
I20260809 05:26:12.896203 19209 tablet_service.cc:1514] Processing DeleteTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:12 UTC) from {username='slave'} at 127.0.0.1:55424
I20260809 05:26:12.896261 19022 tablet_service.cc:1514] Processing DeleteTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:12 UTC) from {username='slave'} at 127.0.0.1:58808
I20260809 05:26:12.896973 18832 tablet_service.cc:1514] Processing DeleteTablet for tablet 8514a2149e8a43f59e86ffcf1a379029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:12 UTC) from {username='slave'} at 127.0.0.1:54566
I20260809 05:26:12.897835 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 18735
I20260809 05:26:12.931630 19359 tablet_replica.cc:330] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: stopping tablet replica
I20260809 05:26:12.934309 19322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:12.899254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.22.1.65:43293)
0809 05:26:12.899437 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:12.899444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:12.923661 (+ 24217us) server_negotiation.cc:407] Connection header received
0809 05:26:12.924352 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:12.924359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:12.924533 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:12.924619 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:12.925358 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.926647 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.928947 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:12.929652 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:12.929755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:12.932381 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:12.932394 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:12.932416 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:12.933307 (+ 891us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:12.933374 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:12.933400 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:12.933494 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:12.933749 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 05:26:12.933868 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:12.935230 19359 raft_consensus.cc:2229] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:12.936398 18913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:12.943193 19359 raft_consensus.cc:2258] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:12.947453 19357 tablet_replica.cc:330] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: stopping tablet replica
I20260809 05:26:12.947968 19357 raft_consensus.cc:2229] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:12.948531 19357 raft_consensus.cc:2258] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:12.950461 19359 ts_tablet_manager.cc:1905] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:12.950886 19357 ts_tablet_manager.cc:1905] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:12.959195 19357 ts_tablet_manager.cc:1918] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:12.959416 19357 log.cc:1196] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/8514a2149e8a43f59e86ffcf1a379029
I20260809 05:26:12.959684 19359 ts_tablet_manager.cc:1918] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:12.959865 19359 log.cc:1196] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/8514a2149e8a43f59e86ffcf1a379029
I20260809 05:26:12.959981 19357 ts_tablet_manager.cc:1939] T 8514a2149e8a43f59e86ffcf1a379029 P 59cd53c7b2aa4c7ab79b0de53c0a6a2e: Deleting consensus metadata
I20260809 05:26:12.960362 19359 ts_tablet_manager.cc:1939] T 8514a2149e8a43f59e86ffcf1a379029 P 813944b913c3410fb84c0a1e9aaf09e1: Deleting consensus metadata
I20260809 05:26:12.962170 18629 catalog_manager.cc:4841] TS 813944b913c3410fb84c0a1e9aaf09e1 (127.22.1.67:39647): tablet 8514a2149e8a43f59e86ffcf1a379029 (table test-table [id=36d0c3eab6054c07b2f0246ff3b0c350]) successfully deleted
I20260809 05:26:12.962158 18624 catalog_manager.cc:4841] TS 59cd53c7b2aa4c7ab79b0de53c0a6a2e (127.22.1.66:41053): tablet 8514a2149e8a43f59e86ffcf1a379029 (table test-table [id=36d0c3eab6054c07b2f0246ff3b0c350]) successfully deleted
I20260809 05:26:13.049883 19360 tablet_replica.cc:330] T 8514a2149e8a43f59e86ffcf1a379029 P e023c145880c404c9c3798890b49c577: stopping tablet replica
I20260809 05:26:13.050241 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 18735
I20260809 05:26:13.075584 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 18929
W20260809 05:26:13.075560 18624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:43293 (error 108)
I20260809 05:26:13.077832 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.076881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54592)
0809 05:26:13.077445 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.077630 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
W20260809 05:26:13.078164 18624 catalog_manager.cc:4584] TS e023c145880c404c9c3798890b49c577 (127.22.1.65:43293): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8514a2149e8a43f59e86ffcf1a379029: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
I20260809 05:26:13.083089 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.076509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58814 (local address 127.22.1.66:41053)
0809 05:26:13.076710 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:13.076716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:13.076736 (+ 20us) server_negotiation.cc:407] Connection header received
0809 05:26:13.076977 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:13.076982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:13.077075 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:13.077213 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:13.077733 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:13.078709 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:13.080045 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:13.080448 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:13.080570 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:13.081711 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:13.081722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:13.081737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:13.082392 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:13.082444 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:13.082466 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:13.082563 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:13.082712 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 05:26:13.082822 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260809 05:26:13.083714 19109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:13.131793 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.131277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54604)
0809 05:26:13.131492 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.131598 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 05:26:13.132848 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.132463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54610)
0809 05:26:13.132636 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.132706 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 05:26:13.173405 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 18929
I20260809 05:26:13.193869 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.193338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54622)
0809 05:26:13.193620 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.193719 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260809 05:26:13.195014 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.194608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54634)
0809 05:26:13.194777 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.194856 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 05:26:13.198441 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 19120
I20260809 05:26:13.205269 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.199273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.22.1.67:39647)
0809 05:26:13.199462 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:13.199467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:13.199488 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:13.199660 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:13.199665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:13.199777 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:13.199870 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:13.200238 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:13.201004 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:13.202402 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:13.202857 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:13.202963 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:13.203952 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:13.203961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:13.203979 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:13.204602 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:13.204652 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:13.204672 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:13.204757 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:13.204885 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 05:26:13.204977 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:26:13.205873 19290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:13.275951 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.275365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54646)
0809 05:26:13.275650 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.275770 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260809 05:26:13.277294 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:13.276728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:43293 (local address 127.0.0.1:54650)
0809 05:26:13.276977 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:13.277072 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:43293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260809 05:26:13.292325 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 19120
I20260809 05:26:13.316427 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 18601
I20260809 05:26:13.317777 18694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:13.425784 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 18601
2026-08-09T05:26:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3175 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:13 dist-test-slave-rmhg krb5kdc[19372](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:13 dist-test-slave-rmhg krb5kdc[19372](info): set up 2 sockets
Aug 09 05:26:13 dist-test-slave-rmhg krb5kdc[19372](info): commencing operation
krb5kdc: starting...
W20260809 05:26:14.660073 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:14 dist-test-slave-rmhg krb5kdc[19372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:14.969884 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41579
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:41579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:15.310222 19395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:15.310534 19395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:15.310652 19395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:15.318116 19395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:15.318255 19395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:15.318307 19395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:15.318387 19395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:15.318446 19395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:15.328723 19395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:41579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:41579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:15.330222 19395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:15.331632 19395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:15.343495 19401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:15.343427 19406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:15.343467 19402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:15.344091 19395 server_base.cc:1034] running on GCE node
I20260809 05:26:15.344954 19395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:15.346448 19395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:15.347652 19395 hybrid_clock.cc:648] HybridClock initialized: now 1786253175347597 us; error 51 us; skew 500 ppm
I20260809 05:26:15.347994 19395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:15 dist-test-slave-rmhg krb5kdc[19372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253175, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:15.351199 19395 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:15.352746 19395 webserver.cc:460] Webserver started at http://127.22.1.126:44527/ using document root <none> and password file <none>
I20260809 05:26:15.353525 19395 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:15.353653 19395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:15.354003 19395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:15.357273 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "0716d6d07c9841da9387ec158bd1b3c6"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "e1dad6a71e3396c843a2909da5b6cb88"
server_key_iv: "4b9b4e42301c037b89217d999481c9d2"
server_key_version: "encryptionkey@0"
I20260809 05:26:15.358112 19395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "0716d6d07c9841da9387ec158bd1b3c6"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "e1dad6a71e3396c843a2909da5b6cb88"
server_key_iv: "4b9b4e42301c037b89217d999481c9d2"
server_key_version: "encryptionkey@0"
I20260809 05:26:15.363739 19395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:15.367370 19413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:15.368708 19395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:15.368877 19395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "0716d6d07c9841da9387ec158bd1b3c6"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "e1dad6a71e3396c843a2909da5b6cb88"
server_key_iv: "4b9b4e42301c037b89217d999481c9d2"
server_key_version: "encryptionkey@0"
I20260809 05:26:15.369061 19395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:15.394918 19395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:15.399629 19395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:15.399928 19395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:15.417474 19395 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:41579
I20260809 05:26:15.417513 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:41579 every 8 connection(s)
I20260809 05:26:15.418865 19395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:15.420941 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 19395
I20260809 05:26:15.421200 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:15.421782 22533 external_mini_cluster.cc:1428] Setting key cbf0fc8d3419bce26988bab78f9ce1a2
I20260809 05:26:15.423400 19492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:15.435019 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: Bootstrap starting.
Aug 09 05:26:15 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:15.438544 19492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:15.439769 19492 log.cc:824] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:15.442727 19492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: No bootstrap required, opened a new log
I20260809 05:26:15.444128 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:15.423803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.22.1.126:41579)
0809 05:26:15.424623 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:15.424635 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:15.424679 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:26:15.426338 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:15.426360 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:15.426747 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:15.427144 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:15.428712 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:15.430044 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:15.431301 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:15.431998 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:15.432196 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:15.436593 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:15.436602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:15.436622 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:15.436675 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:15.439890 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:15.440649 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:15.440656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:15.440664 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:15.440853 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:15.441309 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:15.441313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:15.441315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:15.441893 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:15.442175 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:15.442648 (+ 473us) server_negotiation.cc:299] Negotiation successful
0809 05:26:15.443097 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":332,"threads_started":1}
I20260809 05:26:15.450465 19492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } }
I20260809 05:26:15.450870 19492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:15.450961 19492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0716d6d07c9841da9387ec158bd1b3c6, State: Initialized, Role: FOLLOWER
I20260809 05:26:15.451515 19492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } }
I20260809 05:26:15.451733 19492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:15.451855 19492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:15.452003 19492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:15.454260 19492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } }
I20260809 05:26:15.454737 19492 leader_election.cc:302] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0716d6d07c9841da9387ec158bd1b3c6; no voters:
I20260809 05:26:15.455281 19492 leader_election.cc:288] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:15.455404 19499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:15.456032 19499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [term 1 LEADER]: Becoming Leader. State: Replica: 0716d6d07c9841da9387ec158bd1b3c6, State: Running, Role: LEADER
I20260809 05:26:15.456544 19499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } }
I20260809 05:26:15.457329 19492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:15.459455 19501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0716d6d07c9841da9387ec158bd1b3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } } }
I20260809 05:26:15.459823 19501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:15.461551 19507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:15.461418 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0716d6d07c9841da9387ec158bd1b3c6. Latest consensus state: current_term: 1 leader_uuid: "0716d6d07c9841da9387ec158bd1b3c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0716d6d07c9841da9387ec158bd1b3c6" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 41579 } } }
I20260809 05:26:15.461738 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:15.465993 19507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:15.475617 19507 catalog_manager.cc:1347] Generated new cluster ID: a84a942684854048b5cc644170ccd966
I20260809 05:26:15.475759 19507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:15.506603 19507 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:15.508021 19507 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:15.520417 19507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: Generated new TSK 0
I20260809 05:26:15.521413 19507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.1.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
Entry for principal kudu/127.22.1.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab.
I20260809 05:26:15.590288 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:15.938575 19527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:15.938922 19527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:15.939065 19527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:15.946727 19527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:15.946856 19527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:15.947011 19527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:15.956887 19527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.65.keytab
--principal=kudu/127.22.1.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:15.958374 19527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:15.959839 19527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:15.973134 19541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:15.973222 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:15.973222 19539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:15.978659 19527 server_base.cc:1034] running on GCE node
I20260809 05:26:15.979303 19527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:15.980293 19527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:15.981513 19527 hybrid_clock.cc:648] HybridClock initialized: now 1786253175981457 us; error 44 us; skew 500 ppm
I20260809 05:26:15.981822 19527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:15 dist-test-slave-rmhg krb5kdc[19372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253175, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:15.984897 19527 init.cc:377] Logged in from keytab as kudu/127.22.1.65@KRBTEST.COM (short username kudu)
I20260809 05:26:15.986479 19527 webserver.cc:460] Webserver started at http://127.22.1.65:38205/ using document root <none> and password file <none>
I20260809 05:26:15.987186 19527 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:15.987290 19527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:15.987587 19527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:15.990721 19527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "db9c52b11c22025d0fa67c53aa415bd3"
server_key_iv: "c1cbf64a76510d371c51d5f96a11fed8"
server_key_version: "encryptionkey@0"
I20260809 05:26:15.991488 19527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "db9c52b11c22025d0fa67c53aa415bd3"
server_key_iv: "c1cbf64a76510d371c51d5f96a11fed8"
server_key_version: "encryptionkey@0"
I20260809 05:26:15.997306 19527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:26:16.001264 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.002842 19527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:16.003026 19527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f"
format_stamp: "Formatted at 2026-08-09 05:26:15 on dist-test-slave-rmhg"
server_key: "db9c52b11c22025d0fa67c53aa415bd3"
server_key_iv: "c1cbf64a76510d371c51d5f96a11fed8"
server_key_version: "encryptionkey@0"
I20260809 05:26:16.003238 19527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:16.024766 19527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:16.028831 19527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:16.029094 19527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:16.030126 19527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:16.031505 19527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:16.031584 19527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.031699 19527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:16.031751 19527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.063305 19527 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:46803
I20260809 05:26:16.063370 19711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:46803 every 8 connection(s)
I20260809 05:26:16.064638 19527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:16.071175 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 19527
I20260809 05:26:16.071364 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:16.071967 22533 external_mini_cluster.cc:1428] Setting key f1b6789b36082877258c5679806b71f9
Aug 09 05:26:16 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253175, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.65@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:16.086300 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:16.067127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:42403 (local address 127.22.1.126:41579)
0809 05:26:16.067612 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:16.067618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:16.068666 (+ 1048us) server_negotiation.cc:407] Connection header received
0809 05:26:16.070505 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:16.070510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:16.070578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:16.070676 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:16.072726 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.073746 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.076035 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.076487 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.076611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:16.080698 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:16.080703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:16.080706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:16.080743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:16.083183 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:16.083907 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:16.083912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:16.083915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:16.084042 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:16.084489 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:16.084493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:16.084495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:16.084775 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:16.084913 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:16.085678 (+ 765us) server_negotiation.cc:299] Negotiation successful
0809 05:26:16.085909 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":352,"thread_start_us":214,"threads_started":1}
I20260809 05:26:16.087328 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:16.067790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41579 (local address 127.22.1.65:42403)
0809 05:26:16.068508 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:16.068550 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:16.070241 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:16.070920 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:16.070930 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:16.071717 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:16.072530 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:16.072559 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.073920 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.073924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:16.075841 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:16.075858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.076756 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.076763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:16.077071 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:16.077856 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:16.077893 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:16.080453 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:16.083324 (+ 2871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:16.083334 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:16.083353 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:16.083777 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:16.084152 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:16.084157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:16.084161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:16.084361 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:16.084905 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:16.084914 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:16.085303 (+ 389us) client_negotiation.cc:769] Sending connection context
0809 05:26:16.085568 (+ 265us) client_negotiation.cc:240] Negotiation successful
0809 05:26:16.086143 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":281,"threads_started":1}
I20260809 05:26:16.089097 19712 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41579
I20260809 05:26:16.089422 19712 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:16.090169 19712 heartbeater.cc:511] Master 127.22.1.126:41579 requested a full tablet report, sending...
I20260809 05:26:16.092284 19443 ts_manager.cc:194] Registered new tserver with Master: 5609d9c4d8704113b7c63c4e0e7d8d8f (127.22.1.65:46803)
I20260809 05:26:16.093799 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.65@KRBTEST.COM'} at 127.22.1.65:42403
WARNING: no policy specified for kudu/127.22.1.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
Entry for principal kudu/127.22.1.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab.
I20260809 05:26:16.138397 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:16.489631 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:16.489908 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:16.490036 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:16.497169 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:16.497278 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:16.497452 19721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:16.507313 19721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.66.keytab
--principal=kudu/127.22.1.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:16.508610 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:16.509912 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:16.523622 19728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:16.523591 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:16.523610 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:16.524365 19721 server_base.cc:1034] running on GCE node
I20260809 05:26:16.524984 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:16.525975 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:16.527181 19721 hybrid_clock.cc:648] HybridClock initialized: now 1786253176527132 us; error 38 us; skew 500 ppm
I20260809 05:26:16.527474 19721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:16 dist-test-slave-rmhg krb5kdc[19372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253176, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:16.530643 19721 init.cc:377] Logged in from keytab as kudu/127.22.1.66@KRBTEST.COM (short username kudu)
I20260809 05:26:16.532087 19721 webserver.cc:460] Webserver started at http://127.22.1.66:36431/ using document root <none> and password file <none>
I20260809 05:26:16.532753 19721 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:16.532855 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:16.533123 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:16.536109 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "32f2cd3b1aab472ab8297961a9e2b7af"
format_stamp: "Formatted at 2026-08-09 05:26:16 on dist-test-slave-rmhg"
server_key: "e834132ff4545f29fe3245db6d9e423e"
server_key_iv: "35d98e482bd463ae1fcc7f4c9f73e043"
server_key_version: "encryptionkey@0"
I20260809 05:26:16.536860 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "32f2cd3b1aab472ab8297961a9e2b7af"
format_stamp: "Formatted at 2026-08-09 05:26:16 on dist-test-slave-rmhg"
server_key: "e834132ff4545f29fe3245db6d9e423e"
server_key_iv: "35d98e482bd463ae1fcc7f4c9f73e043"
server_key_version: "encryptionkey@0"
I20260809 05:26:16.541929 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260809 05:26:16.545311 19742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.546576 19721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260809 05:26:16.546721 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "32f2cd3b1aab472ab8297961a9e2b7af"
format_stamp: "Formatted at 2026-08-09 05:26:16 on dist-test-slave-rmhg"
server_key: "e834132ff4545f29fe3245db6d9e423e"
server_key_iv: "35d98e482bd463ae1fcc7f4c9f73e043"
server_key_version: "encryptionkey@0"
I20260809 05:26:16.546893 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:16.571161 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:16.575623 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:16.575876 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:16.576828 19721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:16.578161 19721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:16.578241 19721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.578334 19721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:16.578377 19721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:16.608460 19721 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:33205
I20260809 05:26:16.608515 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:33205 every 8 connection(s)
I20260809 05:26:16.609839 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:16.619853 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 19721
I20260809 05:26:16.620065 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:16.620653 22533 external_mini_cluster.cc:1428] Setting key c21e3905de7e7503d4186ff147b46814
Aug 09 05:26:16 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253176, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.66@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:16.630792 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:16.612430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:58223 (local address 127.22.1.126:41579)
0809 05:26:16.612920 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:16.612927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:16.613959 (+ 1032us) server_negotiation.cc:407] Connection header received
0809 05:26:16.615799 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:16.615804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:16.615882 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:16.615984 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:16.617503 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.618246 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.619766 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.620392 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.620534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:16.624312 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:16.624319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:16.624324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:16.624365 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:16.627421 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:16.628313 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:16.628318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:16.628322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:16.628472 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:16.628904 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:16.628909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:16.628912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:16.629277 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:16.629436 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:16.630227 (+ 791us) server_negotiation.cc:299] Negotiation successful
0809 05:26:16.630445 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":225,"threads_started":1}
I20260809 05:26:16.631827 19901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:16.613020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41579 (local address 127.22.1.66:58223)
0809 05:26:16.613784 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:16.613832 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:16.615522 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:16.616193 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:16.616201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:16.616721 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:16.617358 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:16.617383 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.618378 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.618382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:16.619628 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:16.619640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:16.620606 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:16.620612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:16.620851 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:16.621519 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:16.621542 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:16.624051 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:16.627606 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:16.627615 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:16.627633 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:16.628172 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:16.628563 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:16.628567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:16.628570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:16.628797 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:16.629384 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:16.629391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:16.629814 (+ 423us) client_negotiation.cc:769] Sending connection context
0809 05:26:16.630082 (+ 268us) client_negotiation.cc:240] Negotiation successful
0809 05:26:16.630592 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":311,"threads_started":1}
I20260809 05:26:16.633430 19895 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41579
I20260809 05:26:16.633797 19895 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:16.634732 19895 heartbeater.cc:511] Master 127.22.1.126:41579 requested a full tablet report, sending...
I20260809 05:26:16.636631 19443 ts_manager.cc:194] Registered new tserver with Master: 32f2cd3b1aab472ab8297961a9e2b7af (127.22.1.66:33205)
I20260809 05:26:16.637701 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.66@KRBTEST.COM'} at 127.22.1.66:58223
WARNING: no policy specified for kudu/127.22.1.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
Entry for principal kudu/127.22.1.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab.
I20260809 05:26:16.688190 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:17.032101 19905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:17.032409 19905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:17.032552 19905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:17.040441 19905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:17.040616 19905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:17.040797 19905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:17.054404 19905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.67.keytab
--principal=kudu/127.22.1.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:41579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:17.055760 19905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:17.057111 19905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:17.070979 19911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:17.070941 19912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:17.071424 19905 server_base.cc:1034] running on GCE node
W20260809 05:26:17.071019 19915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:17.072525 19905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:17.073632 19905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:17.074854 19905 hybrid_clock.cc:648] HybridClock initialized: now 1786253177074798 us; error 44 us; skew 500 ppm
I20260809 05:26:17.075220 19905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:17 dist-test-slave-rmhg krb5kdc[19372](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253177, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:17.078681 19905 init.cc:377] Logged in from keytab as kudu/127.22.1.67@KRBTEST.COM (short username kudu)
I20260809 05:26:17.080282 19905 webserver.cc:460] Webserver started at http://127.22.1.67:39649/ using document root <none> and password file <none>
I20260809 05:26:17.080979 19905 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:17.081113 19905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:17.081452 19905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:17.084447 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "b0f93f43721942c18aaa54a51accb6a7"
format_stamp: "Formatted at 2026-08-09 05:26:17 on dist-test-slave-rmhg"
server_key: "a3bce7eb3a1f4a54040f4299c2b03455"
server_key_iv: "cd37b4e489eb956a0cee737463a75078"
server_key_version: "encryptionkey@0"
I20260809 05:26:17.085258 19905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "b0f93f43721942c18aaa54a51accb6a7"
format_stamp: "Formatted at 2026-08-09 05:26:17 on dist-test-slave-rmhg"
server_key: "a3bce7eb3a1f4a54040f4299c2b03455"
server_key_iv: "cd37b4e489eb956a0cee737463a75078"
server_key_version: "encryptionkey@0"
I20260809 05:26:17.090595 19905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:26:17.094077 19923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:17.095420 19905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:17.095584 19905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "b0f93f43721942c18aaa54a51accb6a7"
format_stamp: "Formatted at 2026-08-09 05:26:17 on dist-test-slave-rmhg"
server_key: "a3bce7eb3a1f4a54040f4299c2b03455"
server_key_iv: "cd37b4e489eb956a0cee737463a75078"
server_key_version: "encryptionkey@0"
I20260809 05:26:17.095767 19905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:17.096764 19712 heartbeater.cc:503] Master 127.22.1.126:41579 was elected leader, sending a full tablet report...
I20260809 05:26:17.108743 19905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:17.113095 19905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:17.113425 19905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:17.114455 19905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:17.115909 19905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:17.116004 19905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:17.116130 19905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:17.116185 19905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:17.147641 19905 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:39299
I20260809 05:26:17.147686 20087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:39299 every 8 connection(s)
I20260809 05:26:17.148931 19905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:17.150278 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 19905
I20260809 05:26:17.150465 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:17.151026 22533 external_mini_cluster.cc:1428] Setting key 8996cdc11035607e2e2568b3e89a1e7f
Aug 09 05:26:17 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253177, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.67@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:17.170329 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.153003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:33287 (local address 127.22.1.126:41579)
0809 05:26:17.153522 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.153528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.153547 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:26:17.155796 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.155802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.155904 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.156017 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:17.157977 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.158815 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.160482 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.160883 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.160994 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.164444 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:17.164449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:17.164453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:17.164485 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:17.167534 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.168275 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.168278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.168281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.168397 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.168757 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.168760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.168762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.169025 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:17.169144 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.169897 (+ 753us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.170061 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":1122,"threads_started":1}
I20260809 05:26:17.171401 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.152543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:41579 (local address 127.22.1.67:33287)
0809 05:26:17.153399 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.153467 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.155446 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.156306 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.156316 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.157023 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:17.157784 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.157813 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.158996 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.159002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.160325 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.160343 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.161120 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.161128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.161395 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.162048 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:26:17.162071 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:17.164217 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:17.167681 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:17.167688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:17.167703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:17.168173 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:17.168479 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:26:17.168482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:26:17.168484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:26:17.168669 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:26:17.169196 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:17.169205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:17.169555 (+ 350us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.169833 (+ 278us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.170322 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":320,"threads_started":1}
I20260809 05:26:17.172773 20088 heartbeater.cc:347] Connected to a master server at 127.22.1.126:41579
I20260809 05:26:17.173108 20088 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:17.173913 20088 heartbeater.cc:511] Master 127.22.1.126:41579 requested a full tablet report, sending...
I20260809 05:26:17.175345 19443 ts_manager.cc:194] Registered new tserver with Master: b0f93f43721942c18aaa54a51accb6a7 (127.22.1.67:39299)
I20260809 05:26:17.176136 19443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.1.67@KRBTEST.COM'} at 127.22.1.67:33287
I20260809 05:26:17.179347 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:17.196076 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.183326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.22.1.126:41579)
0809 05:26:17.183538 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.183545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.183882 (+ 337us) server_negotiation.cc:407] Connection header received
0809 05:26:17.184013 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.184018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.184080 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.184170 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:17.185970 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.187040 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.188383 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.188840 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.188939 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.191286 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:17.191291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:17.191294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:17.191327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:17.193690 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.194381 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.194385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.194388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.194514 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.194801 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.194804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.194806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.195129 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:17.195306 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.195592 (+ 286us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.195778 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:26:17.200248 19443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:17.207099 19443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:17.239748 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.219028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.22.1.66:33205)
0809 05:26:17.229917 (+ 10889us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.229928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.231054 (+ 1126us) server_negotiation.cc:407] Connection header received
0809 05:26:17.231180 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.231188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.231464 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.231683 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:17.232367 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.233958 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.236853 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.238376 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.238502 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.238730 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.239276 (+ 546us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.239449 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10664,"thread_start_us":10892,"threads_started":1}
I20260809 05:26:17.239837 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.218803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:33205 (local address 127.0.0.1:47244)
0809 05:26:17.230706 (+ 11903us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.230745 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.230927 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.231768 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.231775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.231803 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:17.232170 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.232182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.234109 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.234114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.236643 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.236660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.238859 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.238864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.239026 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.239065 (+ 39us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.239199 (+ 134us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.239415 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11625,"mutex_wait_us":87,"thread_start_us":12416,"threads_started":1}
I20260809 05:26:17.240960 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.218929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:39299 (local address 127.0.0.1:46646)
0809 05:26:17.231482 (+ 12553us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.231499 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.231616 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.233293 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.233298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.233325 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:17.233698 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.233709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.235718 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.235728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.237972 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.237988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.240130 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.240137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.240327 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.240359 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.240447 (+ 88us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.240585 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12436,"mutex_wait_us":1,"thread_start_us":11373,"threads_started":1}
I20260809 05:26:17.241040 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.219277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.22.1.67:39299)
0809 05:26:17.232413 (+ 13136us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.232423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.232453 (+ 30us) server_negotiation.cc:407] Connection header received
0809 05:26:17.232545 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.232553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.232789 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.232972 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:17.233914 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.235554 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.238150 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.239596 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.239721 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.239974 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.240542 (+ 568us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.240711 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12948,"thread_start_us":12790,"threads_started":1}
I20260809 05:26:17.244984 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc32ada5844c309190e7100faa7837 (DEFAULT_TABLE table=test-table [id=c52684db97284a51b657dfc612d8a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:17.245054 19985 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc32ada5844c309190e7100faa7837 (DEFAULT_TABLE table=test-table [id=c52684db97284a51b657dfc612d8a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:17.245664 20115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.232452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:46803 (local address 127.0.0.1:55122)
0809 05:26:17.237688 (+ 5236us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.237703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.237828 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.238790 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.238795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.238821 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:17.239237 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.239249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.241230 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.241234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.243114 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.243127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.244877 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.244884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.245061 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.245093 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.245226 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.245353 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5119,"thread_start_us":4981,"threads_started":1}
I20260809 05:26:17.245750 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.232060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.22.1.65:46803)
0809 05:26:17.237355 (+ 5295us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.237365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.237825 (+ 460us) server_negotiation.cc:407] Connection header received
0809 05:26:17.237998 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.238005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.238305 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.238536 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:17.239445 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.241035 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.243274 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.244660 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.244791 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.245011 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.245283 (+ 272us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.245440 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5114,"thread_start_us":4904,"threads_started":1}
I20260809 05:26:17.246652 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc32ada5844c309190e7100faa7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:17.246693 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc32ada5844c309190e7100faa7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:17.249800 19614 tablet_service.cc:1467] Processing CreateTablet for tablet 88fc32ada5844c309190e7100faa7837 (DEFAULT_TABLE table=test-table [id=c52684db97284a51b657dfc612d8a176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:17.251348 19614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88fc32ada5844c309190e7100faa7837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:17.259392 20117 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Bootstrap starting.
I20260809 05:26:17.261937 20118 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Bootstrap starting.
I20260809 05:26:17.262540 20117 tablet_bootstrap.cc:654] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:17.263893 20117 log.cc:824] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:17.265569 20120 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Bootstrap starting.
I20260809 05:26:17.266444 20118 tablet_bootstrap.cc:654] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:17.267299 20117 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: No bootstrap required, opened a new log
I20260809 05:26:17.267562 20117 ts_tablet_manager.cc:1397] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260809 05:26:17.267786 20118 log.cc:824] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:17.269099 20120 tablet_bootstrap.cc:654] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:17.270428 20120 log.cc:824] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:17.271092 20118 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: No bootstrap required, opened a new log
I20260809 05:26:17.271399 20118 ts_tablet_manager.cc:1397] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260809 05:26:17.273697 20120 tablet_bootstrap.cc:492] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: No bootstrap required, opened a new log
I20260809 05:26:17.273983 20120 ts_tablet_manager.cc:1397] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260809 05:26:17.275754 20117 raft_consensus.cc:348] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.276157 20117 raft_consensus.cc:374] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:17.276292 20117 raft_consensus.cc:729] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f93f43721942c18aaa54a51accb6a7, State: Initialized, Role: FOLLOWER
I20260809 05:26:17.276965 20117 consensus_queue.cc:260] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.278194 20088 heartbeater.cc:503] Master 127.22.1.126:41579 was elected leader, sending a full tablet report...
I20260809 05:26:17.279026 20117 ts_tablet_manager.cc:1428] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260809 05:26:17.281921 20118 raft_consensus.cc:348] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.282572 20118 raft_consensus.cc:374] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:17.282719 20118 raft_consensus.cc:729] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32f2cd3b1aab472ab8297961a9e2b7af, State: Initialized, Role: FOLLOWER
I20260809 05:26:17.283512 20118 consensus_queue.cc:260] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.284430 20120 raft_consensus.cc:348] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.284821 19895 heartbeater.cc:503] Master 127.22.1.126:41579 was elected leader, sending a full tablet report...
I20260809 05:26:17.284914 20120 raft_consensus.cc:374] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:17.285009 20120 raft_consensus.cc:729] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5609d9c4d8704113b7c63c4e0e7d8d8f, State: Initialized, Role: FOLLOWER
I20260809 05:26:17.285498 20118 ts_tablet_manager.cc:1428] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260809 05:26:17.285753 20120 consensus_queue.cc:260] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.287426 20120 ts_tablet_manager.cc:1428] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260809 05:26:17.317085 19713 tablet.cc:2369] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:17.361857 19897 tablet.cc:2369] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260809 05:26:17.400637 20089 tablet.cc:2369] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:26:17.453712 20123 raft_consensus.cc:482] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:17.454044 20123 raft_consensus.cc:504] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.456003 20123 leader_election.cc:288] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32f2cd3b1aab472ab8297961a9e2b7af (127.22.1.66:33205), 5609d9c4d8704113b7c63c4e0e7d8d8f (127.22.1.65:46803)
I20260809 05:26:17.461730 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.456485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:43719 (local address 127.22.1.66:33205)
0809 05:26:17.456669 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.456674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.456693 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:26:17.456941 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.456947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.457028 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.457132 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:17.457586 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.458330 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.460007 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.460798 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.460924 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.461027 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.461349 (+ 322us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.461478 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:26:17.461731 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.456351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:33205 (local address 127.22.1.67:43719)
0809 05:26:17.456590 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.456611 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.456749 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.457202 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.457206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.457239 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:17.457429 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.457437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.458485 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.458492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.459844 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.459857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.460963 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.460969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.461186 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.461222 (+ 36us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.461307 (+ 85us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.461445 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260809 05:26:17.462858 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.456847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:46803 (local address 127.22.1.67:57309)
0809 05:26:17.457777 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:17.457795 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:17.457913 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:17.458217 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:17.458221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:17.458245 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:17.458631 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.458645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.459677 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.459683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.461000 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:17.461013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.462256 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.462262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:17.462415 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.462449 (+ 34us) client_negotiation.cc:769] Sending connection context
0809 05:26:17.462550 (+ 101us) client_negotiation.cc:240] Negotiation successful
0809 05:26:17.462635 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":645,"threads_started":1}
I20260809 05:26:17.462960 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.457068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:57309 (local address 127.22.1.65:46803)
0809 05:26:17.457283 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.457289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.457881 (+ 592us) server_negotiation.cc:407] Connection header received
0809 05:26:17.458049 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.458053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.458116 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.458186 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:17.458807 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.459534 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.461183 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.462063 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.462184 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.462265 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.462586 (+ 321us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.462723 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:26:17.462982 19829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc32ada5844c309190e7100faa7837" candidate_uuid: "b0f93f43721942c18aaa54a51accb6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" is_pre_election: true
I20260809 05:26:17.463617 19829 raft_consensus.cc:2454] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f93f43721942c18aaa54a51accb6a7 in term 0.
I20260809 05:26:17.463981 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc32ada5844c309190e7100faa7837" candidate_uuid: "b0f93f43721942c18aaa54a51accb6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" is_pre_election: true
I20260809 05:26:17.464354 19928 leader_election.cc:302] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2cd3b1aab472ab8297961a9e2b7af, b0f93f43721942c18aaa54a51accb6a7; no voters:
I20260809 05:26:17.464489 19643 raft_consensus.cc:2454] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0f93f43721942c18aaa54a51accb6a7 in term 0.
I20260809 05:26:17.464751 20123 raft_consensus.cc:2781] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:17.464905 20123 raft_consensus.cc:482] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:17.465019 20123 raft_consensus.cc:3037] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:17.467343 20123 raft_consensus.cc:504] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.467929 20123 leader_election.cc:288] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [CANDIDATE]: Term 1 election: Requested vote from peers 32f2cd3b1aab472ab8297961a9e2b7af (127.22.1.66:33205), 5609d9c4d8704113b7c63c4e0e7d8d8f (127.22.1.65:46803)
I20260809 05:26:17.468400 19829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc32ada5844c309190e7100faa7837" candidate_uuid: "b0f93f43721942c18aaa54a51accb6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32f2cd3b1aab472ab8297961a9e2b7af"
I20260809 05:26:17.468416 19643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88fc32ada5844c309190e7100faa7837" candidate_uuid: "b0f93f43721942c18aaa54a51accb6a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f"
I20260809 05:26:17.468586 19829 raft_consensus.cc:3037] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:17.468595 19643 raft_consensus.cc:3037] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:17.470808 19643 raft_consensus.cc:2454] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f93f43721942c18aaa54a51accb6a7 in term 1.
I20260809 05:26:17.470829 19829 raft_consensus.cc:2454] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0f93f43721942c18aaa54a51accb6a7 in term 1.
I20260809 05:26:17.471190 19928 leader_election.cc:302] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32f2cd3b1aab472ab8297961a9e2b7af, b0f93f43721942c18aaa54a51accb6a7; no voters:
I20260809 05:26:17.471490 20123 raft_consensus.cc:2781] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:17.471926 20123 raft_consensus.cc:686] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 1 LEADER]: Becoming Leader. State: Replica: b0f93f43721942c18aaa54a51accb6a7, State: Running, Role: LEADER
I20260809 05:26:17.472482 20123 consensus_queue.cc:237] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } }
I20260809 05:26:17.477260 19442 catalog_manager.cc:5495] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0f93f43721942c18aaa54a51accb6a7 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "b0f93f43721942c18aaa54a51accb6a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0f93f43721942c18aaa54a51accb6a7" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 39299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 } health_report { overall_health: UNKNOWN } } }
I20260809 05:26:17.526346 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.521269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46650 (local address 127.22.1.67:39299)
0809 05:26:17.521484 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.521489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.521508 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:26:17.521679 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.521684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.521773 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.521900 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:17.522238 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.522944 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.524231 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.524653 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.524746 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.525207 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:17.525813 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:17.525923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.526024 (+ 101us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.526091 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:26:17.534394 19829 raft_consensus.cc:1261] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 1 FOLLOWER]: Refusing update from remote peer b0f93f43721942c18aaa54a51accb6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:17.534394 19643 raft_consensus.cc:1261] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 1 FOLLOWER]: Refusing update from remote peer b0f93f43721942c18aaa54a51accb6a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:17.535282 20123 consensus_queue.cc:1035] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5609d9c4d8704113b7c63c4e0e7d8d8f" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 46803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:17.535604 20128 consensus_queue.cc:1035] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32f2cd3b1aab472ab8297961a9e2b7af" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 33205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:17.547327 20132 mvcc.cc:205] Tried to move back new op lower bound from 7316493015171055616 to 7316493014931513344. Current Snapshot: MvccSnapshot[applied={T|T < 7316493015171055616}]
I20260809 05:26:17.561518 19442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:17.561851 19442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:17.565891 19442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0716d6d07c9841da9387ec158bd1b3c6: Sending DeleteTablet for 3 replicas of tablet 88fc32ada5844c309190e7100faa7837
I20260809 05:26:17.567052 19805 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc32ada5844c309190e7100faa7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:17 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:47244
I20260809 05:26:17.567066 19985 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc32ada5844c309190e7100faa7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:17 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:46646
I20260809 05:26:17.567442 19614 tablet_service.cc:1514] Processing DeleteTablet for tablet 88fc32ada5844c309190e7100faa7837 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:17 UTC) from {username='kudu', principal='kudu/127.22.1.126@KRBTEST.COM'} at 127.0.0.1:55122
I20260809 05:26:17.568651 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 19527
I20260809 05:26:17.578828 20147 tablet_replica.cc:330] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: stopping tablet replica
I20260809 05:26:17.579661 20147 raft_consensus.cc:2229] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:17.580463 20147 raft_consensus.cc:2258] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:17.580744 20146 tablet_replica.cc:330] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: stopping tablet replica
I20260809 05:26:17.581451 20146 raft_consensus.cc:2229] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:26:17.582235 20146 raft_consensus.cc:2258] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:17.583313 20147 ts_tablet_manager.cc:1905] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:17.584573 20146 ts_tablet_manager.cc:1905] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:17.585852 20145 tablet_replica.cc:330] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: stopping tablet replica
I20260809 05:26:17.586442 20145 raft_consensus.cc:2229] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:17.587002 20145 raft_consensus.cc:2258] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:17.591344 20147 ts_tablet_manager.cc:1918] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:17.591622 20147 log.cc:1196] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-0/wal/wals/88fc32ada5844c309190e7100faa7837
I20260809 05:26:17.592321 20147 ts_tablet_manager.cc:1939] T 88fc32ada5844c309190e7100faa7837 P 5609d9c4d8704113b7c63c4e0e7d8d8f: Deleting consensus metadata
I20260809 05:26:17.592953 20146 ts_tablet_manager.cc:1918] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:17.593262 20146 log.cc:1196] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/88fc32ada5844c309190e7100faa7837
I20260809 05:26:17.593870 20146 ts_tablet_manager.cc:1939] T 88fc32ada5844c309190e7100faa7837 P b0f93f43721942c18aaa54a51accb6a7: Deleting consensus metadata
I20260809 05:26:17.594647 19417 catalog_manager.cc:4841] TS 5609d9c4d8704113b7c63c4e0e7d8d8f (127.22.1.65:46803): tablet 88fc32ada5844c309190e7100faa7837 (table test-table [id=c52684db97284a51b657dfc612d8a176]) successfully deleted
I20260809 05:26:17.595808 19419 catalog_manager.cc:4841] TS b0f93f43721942c18aaa54a51accb6a7 (127.22.1.67:39299): tablet 88fc32ada5844c309190e7100faa7837 (table test-table [id=c52684db97284a51b657dfc612d8a176]) successfully deleted
Aug 09 05:26:17 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.65@KRBTEST.COM
I20260809 05:26:17.597441 20145 ts_tablet_manager.cc:1905] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:17.603199 20145 ts_tablet_manager.cc:1918] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:17.603367 20145 log.cc:1196] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/88fc32ada5844c309190e7100faa7837
I20260809 05:26:17.603927 20145 ts_tablet_manager.cc:1939] T 88fc32ada5844c309190e7100faa7837 P 32f2cd3b1aab472ab8297961a9e2b7af: Deleting consensus metadata
I20260809 05:26:17.605662 20114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.570110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.22.1.65:46803)
0809 05:26:17.570313 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.570321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.585301 (+ 14980us) server_negotiation.cc:407] Connection header received
0809 05:26:17.585528 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.585535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.585632 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.585741 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:17.587631 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.588791 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.591015 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.591643 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.591768 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.597879 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:17.597897 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:17.597917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:17.597969 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:17.602033 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.603030 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.603042 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.603046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.603218 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.603823 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.603828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.603831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.604313 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:17.604651 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.604997 (+ 346us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.605282 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:26:17.605844 19417 catalog_manager.cc:4841] TS 32f2cd3b1aab472ab8297961a9e2b7af (127.22.1.66:33205): tablet 88fc32ada5844c309190e7100faa7837 (table test-table [id=c52684db97284a51b657dfc612d8a176]) successfully deleted
I20260809 05:26:17.606647 19704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:17.702224 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 19527
I20260809 05:26:17.740509 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 19721
Aug 09 05:26:17 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.66@KRBTEST.COM
I20260809 05:26:17.757761 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.741334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.22.1.66:33205)
0809 05:26:17.741548 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.741554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.741740 (+ 186us) server_negotiation.cc:407] Connection header received
0809 05:26:17.741819 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.741825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.741925 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.742023 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:17.743477 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.744214 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.745590 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.746044 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.746161 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.751501 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:17.751520 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:17.751539 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:17.751586 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:17.754589 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.755423 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.755432 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.755435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.755592 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.756021 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.756024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.756027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.756470 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:17.756784 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.757134 (+ 350us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.757390 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:26:17.758577 19890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:17.848116 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 19721
I20260809 05:26:17.877926 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 19905
Aug 09 05:26:17 dist-test-slave-rmhg krb5kdc[19372](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.67@KRBTEST.COM
I20260809 05:26:17.896039 20113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:17.878865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.22.1.67:39299)
0809 05:26:17.879068 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:17.879072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:17.879247 (+ 175us) server_negotiation.cc:407] Connection header received
0809 05:26:17.879411 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:17.879417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:17.879523 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:17.879677 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:17.880999 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.882250 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.883521 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:17.884196 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:17.884334 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:17.889040 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:17.889059 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:17.889087 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:17.889139 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:17.892968 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.893817 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.893828 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.893832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.894051 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:17.894412 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:17.894417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:17.894420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:17.894897 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:17.895232 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:17.895387 (+ 155us) server_negotiation.cc:299] Negotiation successful
0809 05:26:17.895612 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 05:26:17.896777 20083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:18.019254 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 19905
I20260809 05:26:18.044195 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 19395
I20260809 05:26:18.045390 19483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:18.154740 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 19395
2026-08-09T05:26:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4727 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-09T05:26:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:18Z Disabled control of system clock
I20260809 05:26:18.247143 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35497
--webserver_interface=127.22.1.126
--webserver_port=0
--builtin_ntp_servers=127.22.1.84:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:18.594842 20164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:18.595175 20164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:18.595299 20164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:18.602767 20164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:18.602891 20164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:18.602932 20164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:18.603013 20164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:18.603075 20164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:18.612298 20164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:18.613674 20164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:18.615464 20164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:18.626679 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:18.626809 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:18.626863 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:18.627882 20164 server_base.cc:1034] running on GCE node
I20260809 05:26:18.628698 20164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:18.630091 20164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:18.631318 20164 hybrid_clock.cc:648] HybridClock initialized: now 1786253178631260 us; error 50 us; skew 500 ppm
I20260809 05:26:18.631649 20164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:18.637699 20164 webserver.cc:460] Webserver started at http://127.22.1.126:45999/ using document root <none> and password file <none>
I20260809 05:26:18.638440 20164 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:18.638538 20164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:18.638829 20164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:18.642242 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "4f7ae6d950ce492d822d20a77e6664d5"
format_stamp: "Formatted at 2026-08-09 05:26:18 on dist-test-slave-rmhg"
server_key: "0a58ecaeb52fad5ec3674e63c05b2c82"
server_key_iv: "04af01d85652c472a80fc9dcda8c7de9"
server_key_version: "encryptionkey@0"
I20260809 05:26:18.643209 20164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "4f7ae6d950ce492d822d20a77e6664d5"
format_stamp: "Formatted at 2026-08-09 05:26:18 on dist-test-slave-rmhg"
server_key: "0a58ecaeb52fad5ec3674e63c05b2c82"
server_key_iv: "04af01d85652c472a80fc9dcda8c7de9"
server_key_version: "encryptionkey@0"
I20260809 05:26:18.648633 20164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260809 05:26:18.652256 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:18.653752 20164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:18.653910 20164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "4f7ae6d950ce492d822d20a77e6664d5"
format_stamp: "Formatted at 2026-08-09 05:26:18 on dist-test-slave-rmhg"
server_key: "0a58ecaeb52fad5ec3674e63c05b2c82"
server_key_iv: "04af01d85652c472a80fc9dcda8c7de9"
server_key_version: "encryptionkey@0"
I20260809 05:26:18.654104 20164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:18.674607 20164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:18.675565 20164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:18.675843 20164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:18.693889 20164 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35497
I20260809 05:26:18.693908 20252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35497 every 8 connection(s)
I20260809 05:26:18.695250 20164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:18.696785 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 20164
I20260809 05:26:18.696965 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:18.697628 22533 external_mini_cluster.cc:1428] Setting key 2072c6849f058774e94d6449ea7106a8
I20260809 05:26:18.699887 20255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:18.711380 20259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:18.699635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.22.1.126:35497)
0809 05:26:18.700633 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:18.700644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:18.700688 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:26:18.702299 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:18.702322 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:18.702784 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:18.703112 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:18.703710 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:18.705083 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:18.706967 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:18.707645 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:18.707781 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:18.708986 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:18.708993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:18.709010 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:18.709649 (+ 639us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:18.709730 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:18.709750 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:18.709846 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:18.710130 (+ 284us) server_negotiation.cc:299] Negotiation successful
0809 05:26:18.710471 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":376,"threads_started":1}
I20260809 05:26:18.712320 20255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: Bootstrap starting.
I20260809 05:26:18.715567 20255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:18.716529 20255 log.cc:824] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:18.719210 20255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: No bootstrap required, opened a new log
I20260809 05:26:18.726725 20255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } }
I20260809 05:26:18.727108 20255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:18.727183 20255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7ae6d950ce492d822d20a77e6664d5, State: Initialized, Role: FOLLOWER
I20260809 05:26:18.727686 20255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } }
I20260809 05:26:18.727877 20255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:18.727980 20255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:18.728097 20255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:18.730214 20255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } }
I20260809 05:26:18.730645 20255 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7ae6d950ce492d822d20a77e6664d5; no voters:
I20260809 05:26:18.731148 20255 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:18.731281 20263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:18.732311 20263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7ae6d950ce492d822d20a77e6664d5, State: Running, Role: LEADER
I20260809 05:26:18.732853 20263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } }
I20260809 05:26:18.733299 20255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:18.735441 20265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7ae6d950ce492d822d20a77e6664d5. Latest consensus state: current_term: 1 leader_uuid: "4f7ae6d950ce492d822d20a77e6664d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } } }
I20260809 05:26:18.735878 20265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:18.736384 20264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7ae6d950ce492d822d20a77e6664d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ae6d950ce492d822d20a77e6664d5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35497 } } }
I20260809 05:26:18.736680 20264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:18.737926 20270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:18.743180 20270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:18.752418 20270 catalog_manager.cc:1347] Generated new cluster ID: c59cea964e8341e68384dc464ee8c065
I20260809 05:26:18.752540 20270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:18.761672 20270 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:18.763756 20270 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:18.773563 20270 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: Generated new TSK 0
I20260809 05:26:18.774638 20270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:18.783879 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.65:0
--local_ip_for_outbound_sockets=127.22.1.65
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--builtin_ntp_servers=127.22.1.84:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260809 05:26:19.135809 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:19.136119 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:19.136233 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:19.143716 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:19.143813 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:19.143946 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.65
I20260809 05:26:19.153359 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.1.65
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.65
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:19.154811 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:19.156806 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:19.170168 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:19.170203 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:19.170135 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:19.170919 20289 server_base.cc:1034] running on GCE node
I20260809 05:26:19.171545 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:19.172518 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:19.173729 20289 hybrid_clock.cc:648] HybridClock initialized: now 1786253179173673 us; error 45 us; skew 500 ppm
I20260809 05:26:19.174068 20289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:19.176273 20289 webserver.cc:460] Webserver started at http://127.22.1.65:35765/ using document root <none> and password file <none>
I20260809 05:26:19.176965 20289 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:19.177083 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:19.177426 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:19.180333 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data/instance:
uuid: "35629e034c7846cab47a6bec73462f52"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "9e3d78620b7195391491e37efba8109a"
server_key_iv: "356f852027a995628b15f4c201b7e7e1"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.181143 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance:
uuid: "35629e034c7846cab47a6bec73462f52"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "9e3d78620b7195391491e37efba8109a"
server_key_iv: "356f852027a995628b15f4c201b7e7e1"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.186719 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:26:19.190501 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.192489 20289 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260809 05:26:19.192680 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal
uuid: "35629e034c7846cab47a6bec73462f52"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "9e3d78620b7195391491e37efba8109a"
server_key_iv: "356f852027a995628b15f4c201b7e7e1"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.192942 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:19.213061 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:19.213956 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:19.214232 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:19.215194 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:19.216734 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:19.216821 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.216912 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:19.216967 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.249366 20289 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.65:44267
I20260809 05:26:19.249385 20477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.65:44267 every 8 connection(s)
I20260809 05:26:19.250705 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/data/info.pb
I20260809 05:26:19.255811 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 20289
I20260809 05:26:19.255973 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-0/wal/instance
I20260809 05:26:19.256402 22533 external_mini_cluster.cc:1428] Setting key b4175248215bbf133ebbc954d1823ab0
I20260809 05:26:19.259426 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.66:0
--local_ip_for_outbound_sockets=127.22.1.66
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--builtin_ntp_servers=127.22.1.84:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:19.263603 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:19.253308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.65:36105 (local address 127.22.1.126:35497)
0809 05:26:19.253761 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:19.253766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:19.254956 (+ 1190us) server_negotiation.cc:407] Connection header received
0809 05:26:19.256861 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:19.256867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:19.256959 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:19.257062 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:19.258448 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.259216 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.260828 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.261287 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.261400 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:19.262775 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:19.262780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:19.262783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:19.262800 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:19.262841 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:19.262852 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:19.262924 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:19.263158 (+ 234us) server_negotiation.cc:299] Negotiation successful
0809 05:26:19.263256 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":230,"threads_started":1}
I20260809 05:26:19.264611 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:19.253889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35497 (local address 127.22.1.65:36105)
0809 05:26:19.254797 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:19.254833 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:19.256581 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:19.257357 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:19.257370 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:19.257508 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:19.258272 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:19.258293 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.259367 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.259371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:19.260680 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:19.260693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.261526 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.261535 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:19.261766 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:19.262438 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:19.262441 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:19.262484 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:19.262582 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:19.262949 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:19.262961 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:19.262967 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:26:19.263128 (+ 161us) client_negotiation.cc:240] Negotiation successful
0809 05:26:19.263547 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":277,"threads_started":1}
I20260809 05:26:19.266103 20478 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35497
I20260809 05:26:19.266427 20478 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:19.267323 20478 heartbeater.cc:511] Master 127.22.1.126:35497 requested a full tablet report, sending...
I20260809 05:26:19.269534 20208 ts_manager.cc:194] Registered new tserver with Master: 35629e034c7846cab47a6bec73462f52 (127.22.1.65:44267)
I20260809 05:26:19.271060 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.65:36105
W20260809 05:26:19.609624 20485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:19.609903 20485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:19.610029 20485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:19.617225 20485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:19.617340 20485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:19.617489 20485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.66
I20260809 05:26:19.626662 20485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.1.66
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.66
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:19.627970 20485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:19.629789 20485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:19.642764 20495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:19.642920 20492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:19.642983 20491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:19.643909 20485 server_base.cc:1034] running on GCE node
I20260809 05:26:19.644578 20485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:19.645637 20485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:19.646863 20485 hybrid_clock.cc:648] HybridClock initialized: now 1786253179646809 us; error 45 us; skew 500 ppm
I20260809 05:26:19.647181 20485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:19.649386 20485 webserver.cc:460] Webserver started at http://127.22.1.66:33231/ using document root <none> and password file <none>
I20260809 05:26:19.650092 20485 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:19.650225 20485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:19.650556 20485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:19.653537 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data/instance:
uuid: "afeb47b5107844a78c2018769a6cce8f"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "28f3d8cfbb28ccfe5911d5f8aa2867ac"
server_key_iv: "acfe2d757ec8d35dd7ba3c1ec7742341"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.654294 20485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance:
uuid: "afeb47b5107844a78c2018769a6cce8f"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "28f3d8cfbb28ccfe5911d5f8aa2867ac"
server_key_iv: "acfe2d757ec8d35dd7ba3c1ec7742341"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.659389 20485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:19.662750 20501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.664077 20485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:26:19.664223 20485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal
uuid: "afeb47b5107844a78c2018769a6cce8f"
format_stamp: "Formatted at 2026-08-09 05:26:19 on dist-test-slave-rmhg"
server_key: "28f3d8cfbb28ccfe5911d5f8aa2867ac"
server_key_iv: "acfe2d757ec8d35dd7ba3c1ec7742341"
server_key_version: "encryptionkey@0"
I20260809 05:26:19.664418 20485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:19.680182 20485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:19.681193 20485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:19.681486 20485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:19.682538 20485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:19.684062 20485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:19.684159 20485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.684271 20485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:19.684341 20485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:19.722229 20485 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.66:43783
I20260809 05:26:19.722275 20660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.66:43783 every 8 connection(s)
I20260809 05:26:19.723650 20485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/data/info.pb
I20260809 05:26:19.731873 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 20485
I20260809 05:26:19.732065 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal/instance
I20260809 05:26:19.732609 22533 external_mini_cluster.cc:1428] Setting key 02d9f2e59102e6d4733bffd280024d86
I20260809 05:26:19.735615 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.1.67:0
--local_ip_for_outbound_sockets=127.22.1.67
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--builtin_ntp_servers=127.22.1.84:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260809 05:26:19.737040 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:19.726386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.66:44551 (local address 127.22.1.126:35497)
0809 05:26:19.726624 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:19.726631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:19.728075 (+ 1444us) server_negotiation.cc:407] Connection header received
0809 05:26:19.729959 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:19.729964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:19.730038 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:19.730132 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:19.731301 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.732356 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.734227 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.734749 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.734872 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:19.736222 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:19.736228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:19.736233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:19.736257 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:19.736295 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:19.736309 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:19.736401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:19.736643 (+ 242us) server_negotiation.cc:299] Negotiation successful
0809 05:26:19.736742 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:26:19.738095 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:19.726912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35497 (local address 127.22.1.66:44551)
0809 05:26:19.727889 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:19.727928 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:19.729680 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:19.730396 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:19.730404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:19.730530 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:19.731137 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:19.731155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.732535 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.732539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:19.734078 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:19.734092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:19.734960 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:19.734965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:19.735175 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:19.735863 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:19.735866 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:19.735928 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:19.736021 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:19.736405 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:19.736416 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:19.736424 (+ 8us) client_negotiation.cc:769] Sending connection context
0809 05:26:19.736607 (+ 183us) client_negotiation.cc:240] Negotiation successful
0809 05:26:19.736914 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":298,"threads_started":1}
I20260809 05:26:19.739539 20664 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35497
I20260809 05:26:19.739856 20664 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:19.740757 20664 heartbeater.cc:511] Master 127.22.1.126:35497 requested a full tablet report, sending...
I20260809 05:26:19.742267 20208 ts_manager.cc:194] Registered new tserver with Master: afeb47b5107844a78c2018769a6cce8f (127.22.1.66:43783)
I20260809 05:26:19.743067 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.66:44551
W20260809 05:26:20.083956 20670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:20.084270 20670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:20.084378 20670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:20.091820 20670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:20.091939 20670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:20.092095 20670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.1.67
I20260809 05:26:20.101505 20670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.1.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.1.67
--webserver_port=0
--tserver_master_addrs=127.22.1.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.1.67
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:20.102936 20670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:20.104873 20670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:20.118758 20676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:20.118841 20680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:20.118728 20677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:20.119354 20670 server_base.cc:1034] running on GCE node
I20260809 05:26:20.120185 20670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:20.121295 20670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:20.122530 20670 hybrid_clock.cc:648] HybridClock initialized: now 1786253180122437 us; error 80 us; skew 500 ppm
I20260809 05:26:20.122846 20670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260809 05:26:20.125368 20670 webserver.cc:460] Webserver started at http://127.22.1.67:37063/ using document root <none> and password file <none>
I20260809 05:26:20.126096 20670 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:20.126207 20670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:20.126613 20670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:20.129773 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data/instance:
uuid: "87d9b6dcc5f640d7afd7d454156d9cd2"
format_stamp: "Formatted at 2026-08-09 05:26:20 on dist-test-slave-rmhg"
server_key: "5c796b9378548d0583155a615155261d"
server_key_iv: "9e55c685bf6fc339573b420560a181cb"
server_key_version: "encryptionkey@0"
I20260809 05:26:20.130640 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance:
uuid: "87d9b6dcc5f640d7afd7d454156d9cd2"
format_stamp: "Formatted at 2026-08-09 05:26:20 on dist-test-slave-rmhg"
server_key: "5c796b9378548d0583155a615155261d"
server_key_iv: "9e55c685bf6fc339573b420560a181cb"
server_key_version: "encryptionkey@0"
I20260809 05:26:20.136399 20670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:20.140332 20687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:20.141779 20670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:20.141949 20670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal
uuid: "87d9b6dcc5f640d7afd7d454156d9cd2"
format_stamp: "Formatted at 2026-08-09 05:26:20 on dist-test-slave-rmhg"
server_key: "5c796b9378548d0583155a615155261d"
server_key_iv: "9e55c685bf6fc339573b420560a181cb"
server_key_version: "encryptionkey@0"
I20260809 05:26:20.142174 20670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:20.163559 20670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:20.164510 20670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:20.164811 20670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:20.165925 20670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260809 05:26:20.167590 20670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260809 05:26:20.167691 20670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:20.167830 20670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260809 05:26:20.167910 20670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:20.199862 20670 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.67:43545
I20260809 05:26:20.199926 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.67:43545 every 8 connection(s)
I20260809 05:26:20.201256 20670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/data/info.pb
I20260809 05:26:20.208230 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 20670
I20260809 05:26:20.208376 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal/instance
I20260809 05:26:20.208842 22533 external_mini_cluster.cc:1428] Setting key 765341b9527ea72fa93f704b7b7f0c37
I20260809 05:26:20.213686 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.203927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:58703 (local address 127.22.1.126:35497)
0809 05:26:20.204115 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.204120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.205568 (+ 1448us) server_negotiation.cc:407] Connection header received
0809 05:26:20.207306 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.207313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.207392 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.207481 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:20.208574 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.209464 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.211125 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.211545 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.211650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.212870 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:20.212875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:20.212878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:20.212894 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:20.212926 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:20.212937 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:20.213009 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.213264 (+ 255us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.213361 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:20.214608 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.204432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35497 (local address 127.22.1.67:58703)
0809 05:26:20.205406 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.205443 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.207032 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.207725 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.207733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.207815 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:26:20.208400 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.208421 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.209593 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.209599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.210974 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.210987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.211765 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.211772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.211980 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.212566 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0809 05:26:20.212568 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:26:20.212606 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0809 05:26:20.212685 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:26:20.213049 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:26:20.213057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:26:20.213063 (+ 6us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.213243 (+ 180us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.213625 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":285,"threads_started":1}
I20260809 05:26:20.215888 20840 heartbeater.cc:347] Connected to a master server at 127.22.1.126:35497
I20260809 05:26:20.216174 20840 heartbeater.cc:464] Registering TS with master...
I20260809 05:26:20.216890 20840 heartbeater.cc:511] Master 127.22.1.126:35497 requested a full tablet report, sending...
I20260809 05:26:20.218272 20208 ts_manager.cc:194] Registered new tserver with Master: 87d9b6dcc5f640d7afd7d454156d9cd2 (127.22.1.67:43545)
I20260809 05:26:20.219033 20208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.22.1.67:58703
I20260809 05:26:20.223745 22533 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260809 05:26:20.233526 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.227944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.22.1.126:35497)
0809 05:26:20.228129 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.228134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.228583 (+ 449us) server_negotiation.cc:407] Connection header received
0809 05:26:20.228751 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.228756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.228831 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.228919 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:20.229381 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.230080 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.231383 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.231792 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.231882 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.232850 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:20.232855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:20.232858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:20.232875 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:20.232906 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:20.232918 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:20.232989 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.233213 (+ 224us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.233296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260809 05:26:20.237540 20208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260809 05:26:20.244599 20208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260809 05:26:20.274078 20478 heartbeater.cc:503] Master 127.22.1.126:35497 was elected leader, sending a full tablet report...
I20260809 05:26:20.276536 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.257103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45350)
0809 05:26:20.268004 (+ 10901us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.268037 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.268209 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.269219 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.269224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.269250 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:20.269655 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.269671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.271422 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.271428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.273530 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.273542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.275645 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.275651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.275807 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.275861 (+ 54us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.275991 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.276187 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10785,"thread_start_us":12612,"threads_started":1}
I20260809 05:26:20.276970 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.256950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.67:43545 (local address 127.0.0.1:56204)
0809 05:26:20.257730 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.257771 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.257933 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.268025 (+ 10092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.268033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.268069 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:20.268478 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.268490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.270961 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.270967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.272835 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.272849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.276193 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.276199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.276378 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.276408 (+ 30us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.276510 (+ 102us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.276638 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"thread_start_us":922,"threads_started":1}
I20260809 05:26:20.277211 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.257455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.22.1.67:43545)
0809 05:26:20.267294 (+ 9839us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.267305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.267348 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:26:20.267476 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.267485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.267778 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.268004 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:20.268743 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.270789 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.273037 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.276013 (+ 2976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.276150 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.276421 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.276592 (+ 171us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.276776 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9632,"thread_start_us":9443,"threads_started":1}
I20260809 05:26:20.277976 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.257276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.22.1.65:44267)
0809 05:26:20.268566 (+ 11290us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.268575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.268607 (+ 32us) server_negotiation.cc:407] Connection header received
0809 05:26:20.268708 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.268716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.268973 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.269192 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:20.269871 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.271288 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.273715 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.275075 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.275202 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.275507 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.277447 (+ 1940us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.277637 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11094,"thread_start_us":12097,"threads_started":1}
I20260809 05:26:20.278209 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.258843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:43783 (local address 127.0.0.1:48054)
0809 05:26:20.269978 (+ 11135us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.269997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.270189 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.270850 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.270855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.270881 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:20.271333 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.271345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.273127 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.273133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.275233 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.275247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.277475 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.277482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.277642 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.277665 (+ 23us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.277746 (+ 81us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.277870 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11011,"mutex_wait_us":4,"thread_start_us":8901,"threads_started":1}
I20260809 05:26:20.278337 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.259397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.22.1.66:43783)
0809 05:26:20.267574 (+ 8177us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.267583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.270091 (+ 2508us) server_negotiation.cc:407] Connection header received
0809 05:26:20.270420 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.270428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.270648 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.270830 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:20.271544 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.272982 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.275483 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.276896 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.277050 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.277335 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.277834 (+ 499us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.278024 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8020,"thread_start_us":7898,"threads_started":1}
I20260809 05:26:20.281687 20375 tablet_service.cc:1467] Processing CreateTablet for tablet ee3a67736bb044cfb49ad76244c2f32c (DEFAULT_TABLE table=test-table [id=5a97536cb4e54a6e82b6a9ffd06ab90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:20.281683 20747 tablet_service.cc:1467] Processing CreateTablet for tablet ee3a67736bb044cfb49ad76244c2f32c (DEFAULT_TABLE table=test-table [id=5a97536cb4e54a6e82b6a9ffd06ab90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:20.281693 20570 tablet_service.cc:1467] Processing CreateTablet for tablet ee3a67736bb044cfb49ad76244c2f32c (DEFAULT_TABLE table=test-table [id=5a97536cb4e54a6e82b6a9ffd06ab90e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260809 05:26:20.283015 20375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3a67736bb044cfb49ad76244c2f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:20.283071 20747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3a67736bb044cfb49ad76244c2f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:20.283066 20570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee3a67736bb044cfb49ad76244c2f32c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:20.293962 20862 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Bootstrap starting.
I20260809 05:26:20.297395 20864 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Bootstrap starting.
I20260809 05:26:20.298059 20863 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Bootstrap starting.
I20260809 05:26:20.298919 20862 tablet_bootstrap.cc:654] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:20.300233 20862 log.cc:824] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:20.301637 20864 tablet_bootstrap.cc:654] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:20.302861 20864 log.cc:824] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:20.302934 20862 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: No bootstrap required, opened a new log
I20260809 05:26:20.303179 20862 ts_tablet_manager.cc:1397] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260809 05:26:20.304109 20863 tablet_bootstrap.cc:654] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:20.305385 20863 log.cc:824] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:20.306098 20864 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: No bootstrap required, opened a new log
I20260809 05:26:20.306354 20864 ts_tablet_manager.cc:1397] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260809 05:26:20.309098 20863 tablet_bootstrap.cc:492] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: No bootstrap required, opened a new log
I20260809 05:26:20.309374 20863 ts_tablet_manager.cc:1397] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.007s
I20260809 05:26:20.311002 20862 raft_consensus.cc:348] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.311367 20862 raft_consensus.cc:374] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:20.311481 20862 raft_consensus.cc:729] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35629e034c7846cab47a6bec73462f52, State: Initialized, Role: FOLLOWER
I20260809 05:26:20.312077 20862 consensus_queue.cc:260] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.314560 20862 ts_tablet_manager.cc:1428] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260809 05:26:20.316636 20864 raft_consensus.cc:348] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.317138 20864 raft_consensus.cc:374] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:20.317303 20864 raft_consensus.cc:729] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afeb47b5107844a78c2018769a6cce8f, State: Initialized, Role: FOLLOWER
I20260809 05:26:20.318118 20864 consensus_queue.cc:260] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.319355 20664 heartbeater.cc:503] Master 127.22.1.126:35497 was elected leader, sending a full tablet report...
I20260809 05:26:20.319682 20863 raft_consensus.cc:348] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.320165 20863 raft_consensus.cc:374] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:20.320278 20863 raft_consensus.cc:729] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87d9b6dcc5f640d7afd7d454156d9cd2, State: Initialized, Role: FOLLOWER
I20260809 05:26:20.320925 20863 consensus_queue.cc:260] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.321602 20864 ts_tablet_manager.cc:1428] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260809 05:26:20.322564 20840 heartbeater.cc:503] Master 127.22.1.126:35497 was elected leader, sending a full tablet report...
I20260809 05:26:20.323050 20863 ts_tablet_manager.cc:1428] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260809 05:26:20.347224 20873 raft_consensus.cc:482] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260809 05:26:20.347558 20873 raft_consensus.cc:504] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.349416 20873 leader_election.cc:288] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35629e034c7846cab47a6bec73462f52 (127.22.1.65:44267), afeb47b5107844a78c2018769a6cce8f (127.22.1.66:43783)
I20260809 05:26:20.354998 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.349760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.66:43783 (local address 127.22.1.67:56177)
0809 05:26:20.349926 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.349952 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.350086 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.350563 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.350571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.350603 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:20.350788 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.350796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.351800 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.351804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.353140 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.353185 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.354268 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.354274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.354425 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.354450 (+ 25us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.354539 (+ 89us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.354677 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260809 05:26:20.355015 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.350017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:56177 (local address 127.22.1.66:43783)
0809 05:26:20.350232 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.350237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.350258 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:20.350329 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.350332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.350401 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.350496 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:20.350963 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.351657 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.353360 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.354104 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.354222 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.354315 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.354605 (+ 290us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.354737 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260809 05:26:20.355830 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.350028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.1.67:49169 (local address 127.22.1.65:44267)
0809 05:26:20.350231 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.350237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.350904 (+ 667us) server_negotiation.cc:407] Connection header received
0809 05:26:20.351052 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.351056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.351141 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.351217 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0809 05:26:20.351780 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.352457 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.354178 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.354952 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.355071 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.355183 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.355468 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.355602 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260809 05:26:20.355906 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.349781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.22.1.67:49169)
0809 05:26:20.350778 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.350792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0809 05:26:20.350911 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:26:20.351255 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:26:20.351259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:26:20.351282 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0809 05:26:20.351593 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.351601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.352609 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.352619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.353973 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:26:20.353986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.355134 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.355141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:26:20.355316 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.355348 (+ 32us) client_negotiation.cc:769] Sending connection context
0809 05:26:20.355423 (+ 75us) client_negotiation.cc:240] Negotiation successful
0809 05:26:20.355548 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":209,"thread_start_us":432,"threads_started":1}
I20260809 05:26:20.356259 20596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3a67736bb044cfb49ad76244c2f32c" candidate_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb47b5107844a78c2018769a6cce8f" is_pre_election: true
I20260809 05:26:20.356854 20596 raft_consensus.cc:2454] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9b6dcc5f640d7afd7d454156d9cd2 in term 0.
I20260809 05:26:20.357024 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3a67736bb044cfb49ad76244c2f32c" candidate_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35629e034c7846cab47a6bec73462f52" is_pre_election: true
I20260809 05:26:20.357658 20404 raft_consensus.cc:2454] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87d9b6dcc5f640d7afd7d454156d9cd2 in term 0.
I20260809 05:26:20.357700 20688 leader_election.cc:302] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87d9b6dcc5f640d7afd7d454156d9cd2, afeb47b5107844a78c2018769a6cce8f; no voters:
I20260809 05:26:20.358172 20873 raft_consensus.cc:2781] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260809 05:26:20.358328 20873 raft_consensus.cc:482] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260809 05:26:20.358444 20873 raft_consensus.cc:3037] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:20.360690 20873 raft_consensus.cc:504] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.361333 20873 leader_election.cc:288] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [CANDIDATE]: Term 1 election: Requested vote from peers 35629e034c7846cab47a6bec73462f52 (127.22.1.65:44267), afeb47b5107844a78c2018769a6cce8f (127.22.1.66:43783)
I20260809 05:26:20.361766 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3a67736bb044cfb49ad76244c2f32c" candidate_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35629e034c7846cab47a6bec73462f52"
I20260809 05:26:20.361853 20596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee3a67736bb044cfb49ad76244c2f32c" candidate_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afeb47b5107844a78c2018769a6cce8f"
I20260809 05:26:20.361958 20404 raft_consensus.cc:3037] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:20.362068 20596 raft_consensus.cc:3037] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:20.364214 20404 raft_consensus.cc:2454] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9b6dcc5f640d7afd7d454156d9cd2 in term 1.
I20260809 05:26:20.364290 20596 raft_consensus.cc:2454] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87d9b6dcc5f640d7afd7d454156d9cd2 in term 1.
I20260809 05:26:20.364686 20691 leader_election.cc:302] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35629e034c7846cab47a6bec73462f52, 87d9b6dcc5f640d7afd7d454156d9cd2; no voters:
I20260809 05:26:20.364964 20873 raft_consensus.cc:2781] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:20.365415 20873 raft_consensus.cc:686] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 87d9b6dcc5f640d7afd7d454156d9cd2, State: Running, Role: LEADER
I20260809 05:26:20.365887 20873 consensus_queue.cc:237] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } }
I20260809 05:26:20.370781 20207 catalog_manager.cc:5495] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87d9b6dcc5f640d7afd7d454156d9cd2 (127.22.1.67). New cstate: current_term: 1 leader_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87d9b6dcc5f640d7afd7d454156d9cd2" member_type: VOTER last_known_addr { host: "127.22.1.67" port: 43545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 } health_report { overall_health: UNKNOWN } } }
I20260809 05:26:20.393666 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.388498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.22.1.67:43545)
0809 05:26:20.388736 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.388742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.388765 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:26:20.388953 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.388957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.389062 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.389218 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:26:20.389510 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.390249 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.391645 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.392055 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.392142 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.392604 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:26:20.393139 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:26:20.393310 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.393349 (+ 39us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.393416 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260809 05:26:20.402215 20404 raft_consensus.cc:1261] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 1 FOLLOWER]: Refusing update from remote peer 87d9b6dcc5f640d7afd7d454156d9cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:20.402215 20596 raft_consensus.cc:1261] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 1 FOLLOWER]: Refusing update from remote peer 87d9b6dcc5f640d7afd7d454156d9cd2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:26:20.403194 20875 consensus_queue.cc:1035] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afeb47b5107844a78c2018769a6cce8f" member_type: VOTER last_known_addr { host: "127.22.1.66" port: 43783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:20.403550 20873 consensus_queue.cc:1035] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35629e034c7846cab47a6bec73462f52" member_type: VOTER last_known_addr { host: "127.22.1.65" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:26:20.413010 20880 mvcc.cc:205] Tried to move back new op lower bound from 7316493026917023744 to 7316493026782494720. Current Snapshot: MvccSnapshot[applied={T|T < 7316493026917023744}]
I20260809 05:26:20.417260 20881 mvcc.cc:205] Tried to move back new op lower bound from 7316493026917023744 to 7316493026782494720. Current Snapshot: MvccSnapshot[applied={T|T < 7316493026917023744}]
I20260809 05:26:20.427799 20207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:20.428084 20207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260809 05:26:20.431677 20207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f7ae6d950ce492d822d20a77e6664d5: Sending DeleteTablet for 3 replicas of tablet ee3a67736bb044cfb49ad76244c2f32c
I20260809 05:26:20.432842 20747 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3a67736bb044cfb49ad76244c2f32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:20 UTC) from {username='slave'} at 127.0.0.1:56204
I20260809 05:26:20.432904 20375 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3a67736bb044cfb49ad76244c2f32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:20 UTC) from {username='slave'} at 127.0.0.1:45350
I20260809 05:26:20.432965 20570 tablet_service.cc:1514] Processing DeleteTablet for tablet ee3a67736bb044cfb49ad76244c2f32c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-09 05:26:20 UTC) from {username='slave'} at 127.0.0.1:48054
I20260809 05:26:20.439625 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 20289
I20260809 05:26:20.443935 20894 tablet_replica.cc:330] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: stopping tablet replica
I20260809 05:26:20.444689 20894 raft_consensus.cc:2229] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 1 LEADER]: Raft consensus shutting down.
I20260809 05:26:20.446202 20895 tablet_replica.cc:330] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: stopping tablet replica
I20260809 05:26:20.445627 20894 raft_consensus.cc:2258] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:20.447273 20897 tablet_replica.cc:330] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: stopping tablet replica
I20260809 05:26:20.447737 20897 raft_consensus.cc:2229] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:20.448379 20897 raft_consensus.cc:2258] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:20.447270 20895 raft_consensus.cc:2229] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260809 05:26:20.450582 20895 raft_consensus.cc:2258] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260809 05:26:20.450714 20897 ts_tablet_manager.cc:1905] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:20.452816 20894 ts_tablet_manager.cc:1905] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:20.455150 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.440935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.22.1.65:44267)
0809 05:26:20.441164 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.441172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.446555 (+ 5383us) server_negotiation.cc:407] Connection header received
0809 05:26:20.446776 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.446782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.446919 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.447039 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:20.447707 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.449011 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.451019 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.451709 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.451815 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.453362 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:20.453375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:20.453395 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:20.454189 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:20.454255 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:20.454284 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:20.454374 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.454648 (+ 274us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.454788 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260809 05:26:20.456620 20468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:20.458246 20897 ts_tablet_manager.cc:1918] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:20.458456 20897 log.cc:1196] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-1/wal/wals/ee3a67736bb044cfb49ad76244c2f32c
I20260809 05:26:20.459092 20897 ts_tablet_manager.cc:1939] T ee3a67736bb044cfb49ad76244c2f32c P afeb47b5107844a78c2018769a6cce8f: Deleting consensus metadata
I20260809 05:26:20.460850 20894 ts_tablet_manager.cc:1918] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260809 05:26:20.461045 20894 log.cc:1196] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Deleting WAL directory at /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786252853919692-22533-0/minicluster-data/ts-2/wal/wals/ee3a67736bb044cfb49ad76244c2f32c
I20260809 05:26:20.461175 20184 catalog_manager.cc:4841] TS afeb47b5107844a78c2018769a6cce8f (127.22.1.66:43783): tablet ee3a67736bb044cfb49ad76244c2f32c (table test-table [id=5a97536cb4e54a6e82b6a9ffd06ab90e]) successfully deleted
I20260809 05:26:20.461629 20894 ts_tablet_manager.cc:1939] T ee3a67736bb044cfb49ad76244c2f32c P 87d9b6dcc5f640d7afd7d454156d9cd2: Deleting consensus metadata
I20260809 05:26:20.463671 20187 catalog_manager.cc:4841] TS 87d9b6dcc5f640d7afd7d454156d9cd2 (127.22.1.67:43545): tablet ee3a67736bb044cfb49ad76244c2f32c (table test-table [id=5a97536cb4e54a6e82b6a9ffd06ab90e]) successfully deleted
I20260809 05:26:20.457883 20895 ts_tablet_manager.cc:1905] T ee3a67736bb044cfb49ad76244c2f32c P 35629e034c7846cab47a6bec73462f52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260809 05:26:20.550629 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 20289
I20260809 05:26:20.577437 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 20485
W20260809 05:26:20.577502 20191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.1.65:44267 (error 108)
I20260809 05:26:20.579589 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.579031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45356)
0809 05:26:20.579232 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.579381 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260809 05:26:20.579828 20191 catalog_manager.cc:4584] TS 35629e034c7846cab47a6bec73462f52 (127.22.1.65:44267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee3a67736bb044cfb49ad76244c2f32c: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
I20260809 05:26:20.586086 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.579035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.22.1.66:43783)
0809 05:26:20.579232 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.579239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.579260 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:26:20.579328 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.579333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.579445 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.579557 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:20.580155 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.580958 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.582884 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.583317 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.583424 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.584629 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:20.584642 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:20.584658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:20.585338 (+ 680us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:20.585393 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:20.585414 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:20.585503 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.585662 (+ 159us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.585761 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260809 05:26:20.586771 20654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:20.633726 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.633092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45370)
0809 05:26:20.633400 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.633524 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260809 05:26:20.635083 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.634608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45384)
0809 05:26:20.634796 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.634908 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260809 05:26:20.677523 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 20485
I20260809 05:26:20.695807 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.695230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45390)
0809 05:26:20.695525 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.695620 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260809 05:26:20.696925 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.696523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45396)
0809 05:26:20.696714 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.696790 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260809 05:26:20.712767 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 20670
I20260809 05:26:20.721475 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.713639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.22.1.67:43545)
0809 05:26:20.713839 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:20.713845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:20.713868 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:26:20.714042 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:20.714048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:20.714162 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:20.714280 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:20.714728 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.715962 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.717465 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:20.718167 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:20.718319 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:20.719396 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:20.719410 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:20.719433 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:20.720349 (+ 916us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:20.720429 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:20.720461 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:20.720576 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:20.720742 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 05:26:20.720865 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:26:20.722307 20833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:20.777732 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.777081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45402)
0809 05:26:20.777354 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.777523 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260809 05:26:20.778909 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.778428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45412)
0809 05:26:20.778676 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.778769 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260809 05:26:20.843714 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 20670
I20260809 05:26:20.869485 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 20164
I20260809 05:26:20.870563 20247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:20.980329 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:20.979640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.65:44267 (local address 127.0.0.1:45426)
0809 05:26:20.980017 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0809 05:26:20.980124 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.65:44267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260809 05:26:20.980350 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 20164
2026-08-09T05:26:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2831 ms)
[----------] 6 tests from EncryptionPolicyTest (22927 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:21 dist-test-slave-rmhg krb5kdc[20909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:21 dist-test-slave-rmhg krb5kdc[20909](info): set up 2 sockets
Aug 09 05:26:21 dist-test-slave-rmhg krb5kdc[20909](info): commencing operation
krb5kdc: starting...
W20260809 05:26:22.218034 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:22 dist-test-slave-rmhg krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:22.536818 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:22.887138 20926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:22.887475 20926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:22.887629 20926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:22.894904 20926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:22.895047 20926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:22.895133 20926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:22.895216 20926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:22.895277 20926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:22.905304 20926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33599
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:22.906689 20926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:22.908058 20926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:22.919926 20934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:22.919905 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:22.919961 20936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:22.921041 20926 server_base.cc:1034] running on GCE node
I20260809 05:26:22.921914 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:22.923421 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:22.924639 20926 hybrid_clock.cc:648] HybridClock initialized: now 1786253182924582 us; error 53 us; skew 500 ppm
I20260809 05:26:22.925004 20926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:22 dist-test-slave-rmhg krb5kdc[20909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:22.928220 20926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:22.929844 20926 webserver.cc:460] Webserver started at http://127.0.0.1:45849/ using document root <none> and password file <none>
I20260809 05:26:22.930517 20926 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:22.930620 20926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:22.930876 20926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:22.933915 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "9b6010cc110b4d31aa210181426e50b0"
format_stamp: "Formatted at 2026-08-09 05:26:22 on dist-test-slave-rmhg"
server_key: "ba31da8c4fda1c963545199c4ced6f2e"
server_key_iv: "4f3f8654ec9628df780fdb5b2e65d27e"
server_key_version: "encryptionkey@0"
I20260809 05:26:22.934653 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "9b6010cc110b4d31aa210181426e50b0"
format_stamp: "Formatted at 2026-08-09 05:26:22 on dist-test-slave-rmhg"
server_key: "ba31da8c4fda1c963545199c4ced6f2e"
server_key_iv: "4f3f8654ec9628df780fdb5b2e65d27e"
server_key_version: "encryptionkey@0"
I20260809 05:26:22.939781 20926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:22.943322 20950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:22.944734 20926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:22.944886 20926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "9b6010cc110b4d31aa210181426e50b0"
format_stamp: "Formatted at 2026-08-09 05:26:22 on dist-test-slave-rmhg"
server_key: "ba31da8c4fda1c963545199c4ced6f2e"
server_key_iv: "4f3f8654ec9628df780fdb5b2e65d27e"
server_key_version: "encryptionkey@0"
I20260809 05:26:22.945055 20926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:22.957226 20926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:22.961426 20926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:22.961719 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:22.979573 20926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33599
I20260809 05:26:22.979607 21025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33599 every 8 connection(s)
I20260809 05:26:22.980945 20926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:22.984758 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:22.988341 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 20926
I20260809 05:26:22.988575 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:22.989454 22533 external_mini_cluster.cc:1428] Setting key 901bf0a665f036bc1f6f33b666c74504
I20260809 05:26:22.998116 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: Bootstrap starting.
I20260809 05:26:23.002084 21026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:23.003492 21026 log.cc:824] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:26:22 dist-test-slave-rmhg krb5kdc[20909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 05:26:23.006719 21026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: No bootstrap required, opened a new log
I20260809 05:26:23.013824 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:22.991474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49171 (local address 127.0.0.1:33599)
0809 05:26:22.992244 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:22.992256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:22.992298 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:26:22.993924 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:22.993950 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:22.994374 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:22.994712 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:22.996576 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:22.998004 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:22.999926 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:23.000690 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:23.000830 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:23.006192 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:23.006201 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:23.006218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:23.006256 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:23.009430 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:23.010286 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:23.010292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:23.010299 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:23.010490 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:23.010975 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:23.010979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:23.010981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:23.011534 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:23.011849 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:23.012377 (+ 528us) server_negotiation.cc:299] Negotiation successful
0809 05:26:23.012721 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":357,"threads_started":1}
I20260809 05:26:23.014436 21026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } }
I20260809 05:26:23.014789 21026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:23.014945 21026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b6010cc110b4d31aa210181426e50b0, State: Initialized, Role: FOLLOWER
I20260809 05:26:23.015547 21026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } }
I20260809 05:26:23.015766 21026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:23.015899 21026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:23.016058 21026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:23.017961 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:23.018579 21026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } }
I20260809 05:26:23.019047 21026 leader_election.cc:302] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b6010cc110b4d31aa210181426e50b0; no voters:
I20260809 05:26:23.019599 21026 leader_election.cc:288] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:23.019781 21031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:23.021646 21026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:23.025197 21031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [term 1 LEADER]: Becoming Leader. State: Replica: 9b6010cc110b4d31aa210181426e50b0, State: Running, Role: LEADER
I20260809 05:26:23.025985 21031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } }
I20260809 05:26:23.030773 21034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b6010cc110b4d31aa210181426e50b0. Latest consensus state: current_term: 1 leader_uuid: "9b6010cc110b4d31aa210181426e50b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } } }
I20260809 05:26:23.031154 21034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:23.033423 21033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b6010cc110b4d31aa210181426e50b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b6010cc110b4d31aa210181426e50b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33599 } } }
I20260809 05:26:23.033727 21033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0 [sys.catalog]: This master's current role is: LEADER
W20260809 05:26:23.039332 21063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:26:23.039484 21063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:26:23.039672 21061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:23.041973 21061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:23.047004 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:23.028929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44173 (local address 127.0.0.1:33599)
0809 05:26:23.029108 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:23.029114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:23.029261 (+ 147us) server_negotiation.cc:407] Connection header received
0809 05:26:23.029485 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:23.029489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:23.029554 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:23.029636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:23.031425 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:23.032681 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:23.036663 (+ 3982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:23.037513 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:23.037655 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:23.040830 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:23.040835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:23.040838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:23.040877 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:23.043867 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:23.044738 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:23.044742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:23.044744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:23.044867 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:23.045410 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:23.045414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:23.045416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:23.045779 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:23.046097 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:23.046474 (+ 377us) server_negotiation.cc:299] Negotiation successful
0809 05:26:23.046649 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:26:23.051994 21061 catalog_manager.cc:1347] Generated new cluster ID: 00cc257199fa48dfbe26f58296b04ba5
I20260809 05:26:23.052124 21061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:23.063534 21061 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:23.065420 21061 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:23.073330 21061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b6010cc110b4d31aa210181426e50b0: Generated new TSK 0
I20260809 05:26:23.074503 21061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:23.091507 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 20926
I20260809 05:26:23.093174 21020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:23.183976 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 20926
W20260809 05:26:23.205571 20906 connection.cc:535] client connection to 127.0.0.1:33599 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33599): Connection reset by peer (error 104)
2026-08-09T05:26:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2192 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:23 dist-test-slave-rmhg krb5kdc[21071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:23 dist-test-slave-rmhg krb5kdc[21071](info): set up 2 sockets
Aug 09 05:26:23 dist-test-slave-rmhg krb5kdc[21071](info): commencing operation
krb5kdc: starting...
W20260809 05:26:24.414041 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.001s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:24 dist-test-slave-rmhg krb5kdc[21071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:24.730026 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:25.074036 21090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:25.074331 21090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:25.074431 21090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:25.081730 21090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:25.081866 21090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:25.081921 21090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:25.081983 21090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:25.082028 21090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:25.092214 21090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:25.093606 21090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:25.094923 21090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:25.106492 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:25.106438 21098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:25.106412 21100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:25.107335 21090 server_base.cc:1034] running on GCE node
I20260809 05:26:25.108129 21090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:25.109663 21090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:25.110888 21090 hybrid_clock.cc:648] HybridClock initialized: now 1786253185110827 us; error 50 us; skew 500 ppm
I20260809 05:26:25.111232 21090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:25 dist-test-slave-rmhg krb5kdc[21071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:25.117612 21090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:25.119246 21090 webserver.cc:460] Webserver started at http://127.0.0.1:42607/ using document root <none> and password file <none>
I20260809 05:26:25.120062 21090 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:25.120204 21090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:25.120525 21090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:25.123986 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "5a24efab73b24568879a62bf6cede4b4"
format_stamp: "Formatted at 2026-08-09 05:26:25 on dist-test-slave-rmhg"
server_key: "bf666ece9bbdfdef5cfc787b9c9d816b"
server_key_iv: "391ae5400b44abd18658acb0d2f21d6a"
server_key_version: "encryptionkey@0"
I20260809 05:26:25.124845 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "5a24efab73b24568879a62bf6cede4b4"
format_stamp: "Formatted at 2026-08-09 05:26:25 on dist-test-slave-rmhg"
server_key: "bf666ece9bbdfdef5cfc787b9c9d816b"
server_key_iv: "391ae5400b44abd18658acb0d2f21d6a"
server_key_version: "encryptionkey@0"
I20260809 05:26:25.130375 21090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260809 05:26:25.133980 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:25.135422 21090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:25.135620 21090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "5a24efab73b24568879a62bf6cede4b4"
format_stamp: "Formatted at 2026-08-09 05:26:25 on dist-test-slave-rmhg"
server_key: "bf666ece9bbdfdef5cfc787b9c9d816b"
server_key_iv: "391ae5400b44abd18658acb0d2f21d6a"
server_key_version: "encryptionkey@0"
I20260809 05:26:25.135811 21090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:25.158825 21090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:25.163817 21090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:25.164148 21090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:25.182221 21090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45443
I20260809 05:26:25.182271 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45443 every 8 connection(s)
I20260809 05:26:25.183607 21090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:25.187464 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:25.191665 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21090
I20260809 05:26:25.191859 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:25.192409 22533 external_mini_cluster.cc:1428] Setting key 954c44e4b197d7c576d65251b6b7ab41
I20260809 05:26:25.200929 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4: Bootstrap starting.
I20260809 05:26:25.205101 21182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:25.206363 21182 log.cc:824] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:25.206844 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:25.194368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39527 (local address 127.0.0.1:45443)
0809 05:26:25.195178 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:25.195190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:25.195232 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:26:25.196785 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:25.196809 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:25.197309 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:25.197675 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:25.198305 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:25.199706 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:25.201729 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:25.202791 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:25.202957 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:25.204345 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:25.204353 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:25.204371 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:25.204960 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:25.205062 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:25.205094 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:25.205316 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:25.205534 (+ 218us) server_negotiation.cc:299] Negotiation successful
0809 05:26:25.205729 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":379,"threads_started":1}
I20260809 05:26:25.209412 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4: No bootstrap required, opened a new log
I20260809 05:26:25.210698 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:25.218770 21182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } }
I20260809 05:26:25.219180 21182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:25.219290 21182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a24efab73b24568879a62bf6cede4b4, State: Initialized, Role: FOLLOWER
I20260809 05:26:25.219915 21182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } }
I20260809 05:26:25.220155 21182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:25.220295 21182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:25.220445 21182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:25.223001 21182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } }
I20260809 05:26:25.223340 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:25.216051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35577 (local address 127.0.0.1:45443)
0809 05:26:25.216206 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:25.216210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:25.217622 (+ 1412us) server_negotiation.cc:407] Connection header received
0809 05:26:25.217808 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:25.217812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:25.217876 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:25.217967 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:25.218485 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:25.219297 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:25.220839 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:25.221358 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:25.221478 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:25.222600 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:25.222605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:25.222609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:25.222626 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:25.222663 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:25.222679 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:25.222774 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:25.222990 (+ 216us) server_negotiation.cc:299] Negotiation successful
0809 05:26:25.223047 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:26:25.223614 21182 leader_election.cc:302] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a24efab73b24568879a62bf6cede4b4; no voters:
I20260809 05:26:25.224299 21182 leader_election.cc:288] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:25.224453 21194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:25.225592 21194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a24efab73b24568879a62bf6cede4b4, State: Running, Role: LEADER
I20260809 05:26:25.226110 21194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } }
I20260809 05:26:25.226743 21182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:25.228821 21195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a24efab73b24568879a62bf6cede4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } } }
I20260809 05:26:25.228796 21196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a24efab73b24568879a62bf6cede4b4. Latest consensus state: current_term: 1 leader_uuid: "5a24efab73b24568879a62bf6cede4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a24efab73b24568879a62bf6cede4b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45443 } } }
I20260809 05:26:25.229331 21195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:25.229331 21196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:25.231319 21200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:25.236260 21200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:25.246182 21200 catalog_manager.cc:1347] Generated new cluster ID: 1a8efac69249486b8a3e193a6814724f
I20260809 05:26:25.246304 21200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:25.256004 21200 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:25.257319 21200 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:25.271873 21200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a24efab73b24568879a62bf6cede4b4: Generated new TSK 0
I20260809 05:26:25.272831 21200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:25.288686 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21090
W20260809 05:26:25.290611 21175 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39527
W20260809 05:26:25.291416 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:25.291589 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21090
W20260809 05:26:25.315233 21067 connection.cc:535] client connection to 127.0.0.1:45443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45443): Connection reset by peer (error 104)
2026-08-09T05:26:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2109 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:25 dist-test-slave-rmhg krb5kdc[21225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:25 dist-test-slave-rmhg krb5kdc[21225](info): set up 2 sockets
Aug 09 05:26:25 dist-test-slave-rmhg krb5kdc[21225](info): commencing operation
krb5kdc: starting...
W20260809 05:26:26.516014 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:26 dist-test-slave-rmhg krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:26.833492 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41455
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:27.180814 21245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:27.181121 21245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:27.181263 21245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:27.188545 21245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:27.188666 21245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:27.188712 21245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:27.188771 21245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:27.188817 21245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:27.198921 21245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:27.200253 21245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:27.201646 21245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:27.213554 21251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:27.213526 21256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:27.213589 21252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:27.214327 21245 server_base.cc:1034] running on GCE node
I20260809 05:26:27.215015 21245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:27.216362 21245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:27.217571 21245 hybrid_clock.cc:648] HybridClock initialized: now 1786253187217478 us; error 47 us; skew 500 ppm
I20260809 05:26:27.217978 21245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:27 dist-test-slave-rmhg krb5kdc[21225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:27.221529 21245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:27.223016 21245 webserver.cc:460] Webserver started at http://127.0.0.1:37973/ using document root <none> and password file <none>
I20260809 05:26:27.223680 21245 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:27.223805 21245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:27.224100 21245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:27.227142 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "1f2bb4c1e1cb4713bfd919f572ef2311"
format_stamp: "Formatted at 2026-08-09 05:26:27 on dist-test-slave-rmhg"
server_key: "55a6cfeb8b2abd36c7468c09dfe181bb"
server_key_iv: "08c82b0ddb5a5815ffac5e26dacbd4c4"
server_key_version: "encryptionkey@0"
I20260809 05:26:27.227919 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "1f2bb4c1e1cb4713bfd919f572ef2311"
format_stamp: "Formatted at 2026-08-09 05:26:27 on dist-test-slave-rmhg"
server_key: "55a6cfeb8b2abd36c7468c09dfe181bb"
server_key_iv: "08c82b0ddb5a5815ffac5e26dacbd4c4"
server_key_version: "encryptionkey@0"
I20260809 05:26:27.232954 21245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260809 05:26:27.236346 21265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:27.237727 21245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:27.237895 21245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "1f2bb4c1e1cb4713bfd919f572ef2311"
format_stamp: "Formatted at 2026-08-09 05:26:27 on dist-test-slave-rmhg"
server_key: "55a6cfeb8b2abd36c7468c09dfe181bb"
server_key_iv: "08c82b0ddb5a5815ffac5e26dacbd4c4"
server_key_version: "encryptionkey@0"
I20260809 05:26:27.238075 21245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:27.244349 21245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:27.248806 21245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:27.249106 21245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:27.266678 21245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41455
I20260809 05:26:27.266722 21338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41455 every 8 connection(s)
I20260809 05:26:27.268047 21245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:27.271705 21339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:27.275346 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21245
I20260809 05:26:27.275542 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:27.276090 22533 external_mini_cluster.cc:1428] Setting key 7f8ce5c1a100971ced6ca623f5cbab91
I20260809 05:26:27.283964 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:27.277979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57223 (local address 127.0.0.1:41455)
0809 05:26:27.278756 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:27.278768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:27.278812 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:26:27.280425 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:27.280449 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:27.280815 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:27.281203 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:27.282337 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:27.282343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:27.282358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:27.282373 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:27.282438 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:27.282459 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:27.282637 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:27.282766 (+ 129us) server_negotiation.cc:299] Negotiation successful
0809 05:26:27.282951 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":347,"threads_started":1}
I20260809 05:26:27.284642 21339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311: Bootstrap starting.
I20260809 05:26:27.288103 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:27.288309 21339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:27.289390 21339 log.cc:824] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:27.292641 21339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311: No bootstrap required, opened a new log
I20260809 05:26:27.295147 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:27.292689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40559 (local address 127.0.0.1:41455)
0809 05:26:27.292862 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:27.292869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:27.293283 (+ 414us) server_negotiation.cc:407] Connection header received
0809 05:26:27.293466 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:27.293472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:27.293525 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:27.293611 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:27.294497 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:27.294503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:27.294508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:27.294525 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:27.294568 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:27.294584 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:27.294690 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:27.294831 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 05:26:27.294885 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:26:27.300489 21339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } }
I20260809 05:26:27.300868 21339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:27.300992 21339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f2bb4c1e1cb4713bfd919f572ef2311, State: Initialized, Role: FOLLOWER
I20260809 05:26:27.301656 21339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } }
I20260809 05:26:27.301954 21339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:27.302081 21339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:27.302209 21339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:27.304337 21339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } }
I20260809 05:26:27.304843 21339 leader_election.cc:302] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f2bb4c1e1cb4713bfd919f572ef2311; no voters:
I20260809 05:26:27.305411 21339 leader_election.cc:288] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:27.305541 21356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:27.306197 21356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [term 1 LEADER]: Becoming Leader. State: Replica: 1f2bb4c1e1cb4713bfd919f572ef2311, State: Running, Role: LEADER
I20260809 05:26:27.306744 21356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } }
I20260809 05:26:27.307623 21339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:27.309533 21358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f2bb4c1e1cb4713bfd919f572ef2311. Latest consensus state: current_term: 1 leader_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } } }
I20260809 05:26:27.309564 21357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f2bb4c1e1cb4713bfd919f572ef2311" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41455 } } }
I20260809 05:26:27.309985 21358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:27.309985 21357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:27.311692 21365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:27.317355 21365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:27.327077 21365 catalog_manager.cc:1347] Generated new cluster ID: ba1dbbbca84047709c5ce15986490693
I20260809 05:26:27.327215 21365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:27.342474 21365 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:27.342944 21365 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:27.350471 21365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f2bb4c1e1cb4713bfd919f572ef2311: Generated new TSK 0
I20260809 05:26:27.351402 21365 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:27.367942 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21245
W20260809 05:26:27.369143 21334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57223
W20260809 05:26:27.370036 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:27.370172 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21245
2026-08-09T05:26:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:27 dist-test-slave-rmhg krb5kdc[21389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:27 dist-test-slave-rmhg krb5kdc[21389](info): set up 2 sockets
Aug 09 05:26:27 dist-test-slave-rmhg krb5kdc[21389](info): commencing operation
krb5kdc: starting...
W20260809 05:26:28.596005 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:28 dist-test-slave-rmhg krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:28.912478 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:29.258386 21405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:29.258733 21405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:29.258881 21405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:29.266202 21405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:29.266323 21405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:29.266400 21405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:29.266481 21405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:29.266541 21405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:29.276244 21405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:29.277627 21405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:29.279014 21405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:29.291385 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:29.291477 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:29.291446 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:29.292109 21405 server_base.cc:1034] running on GCE node
I20260809 05:26:29.292966 21405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:29.294516 21405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:29.295743 21405 hybrid_clock.cc:648] HybridClock initialized: now 1786253189295689 us; error 53 us; skew 500 ppm
I20260809 05:26:29.296110 21405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:29 dist-test-slave-rmhg krb5kdc[21389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:29.299293 21405 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:29.300889 21405 webserver.cc:460] Webserver started at http://127.0.0.1:37095/ using document root <none> and password file <none>
I20260809 05:26:29.301641 21405 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:29.301765 21405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:29.302053 21405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:29.305130 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "390024bd15214eee904a246e3c112144"
format_stamp: "Formatted at 2026-08-09 05:26:29 on dist-test-slave-rmhg"
server_key: "05266963ac048e0a795efe8b2fc95b78"
server_key_iv: "7ed783a728740e45c5d59955956aff8a"
server_key_version: "encryptionkey@0"
I20260809 05:26:29.306015 21405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "390024bd15214eee904a246e3c112144"
format_stamp: "Formatted at 2026-08-09 05:26:29 on dist-test-slave-rmhg"
server_key: "05266963ac048e0a795efe8b2fc95b78"
server_key_iv: "7ed783a728740e45c5d59955956aff8a"
server_key_version: "encryptionkey@0"
I20260809 05:26:29.311650 21405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 05:26:29.315210 21422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:29.316550 21405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:29.316737 21405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "390024bd15214eee904a246e3c112144"
format_stamp: "Formatted at 2026-08-09 05:26:29 on dist-test-slave-rmhg"
server_key: "05266963ac048e0a795efe8b2fc95b78"
server_key_iv: "7ed783a728740e45c5d59955956aff8a"
server_key_version: "encryptionkey@0"
I20260809 05:26:29.316924 21405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:29.343659 21405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:29.347904 21405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:29.348181 21405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:29.366122 21405 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43751
I20260809 05:26:29.366146 21491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43751 every 8 connection(s)
I20260809 05:26:29.367667 21405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:29.371462 21492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:29.374727 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21405
I20260809 05:26:29.374907 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:29.375401 22533 external_mini_cluster.cc:1428] Setting key 2f0c4349862ea4205374d4a105e37152
I20260809 05:26:29.383989 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144: Bootstrap starting.
I20260809 05:26:29.387375 21492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:29.388441 21492 log.cc:824] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144: Log is configured to *not* fsync() on all Append() calls
Aug 09 05:26:29 dist-test-slave-rmhg krb5kdc[21389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 05:26:29.391746 21492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144: No bootstrap required, opened a new log
I20260809 05:26:29.399513 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:29.377317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39187 (local address 127.0.0.1:43751)
0809 05:26:29.378104 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:29.378122 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:29.378167 (+ 45us) server_negotiation.cc:407] Connection header received
0809 05:26:29.379858 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:29.379884 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:29.380369 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:29.380735 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:29.382637 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:29.384029 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:29.385468 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:29.386197 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:29.386384 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:29.390659 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:29.390671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:29.390692 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:29.390740 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:29.394647 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:29.395428 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:29.395437 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:29.395450 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:29.395664 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:29.396033 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:29.396038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:29.396041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:29.396766 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:29.397082 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:29.397485 (+ 403us) server_negotiation.cc:299] Negotiation successful
0809 05:26:29.397977 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260809 05:26:29.402288 21492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } }
I20260809 05:26:29.402705 21492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:29.402802 21492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 390024bd15214eee904a246e3c112144, State: Initialized, Role: FOLLOWER
I20260809 05:26:29.403465 21492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } }
I20260809 05:26:29.403800 21492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:29.403949 21492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:29.404047 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:29.404132 21492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:29.406549 21492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } }
I20260809 05:26:29.407114 21492 leader_election.cc:302] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 390024bd15214eee904a246e3c112144; no voters:
I20260809 05:26:29.407757 21492 leader_election.cc:288] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:29.407938 21503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:29.409927 21492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:29.414821 21503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [term 1 LEADER]: Becoming Leader. State: Replica: 390024bd15214eee904a246e3c112144, State: Running, Role: LEADER
I20260809 05:26:29.415550 21503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } }
I20260809 05:26:29.418666 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "390024bd15214eee904a246e3c112144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } } }
I20260809 05:26:29.419081 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:29.419535 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 390024bd15214eee904a246e3c112144. Latest consensus state: current_term: 1 leader_uuid: "390024bd15214eee904a246e3c112144" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "390024bd15214eee904a246e3c112144" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43751 } } }
I20260809 05:26:29.419817 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:29.427947 21526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:29.433832 21526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:29.444025 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:29.418572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55045 (local address 127.0.0.1:43751)
0809 05:26:29.418765 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:29.418771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:29.429885 (+ 11114us) server_negotiation.cc:407] Connection header received
0809 05:26:29.430217 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:29.430227 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:29.430316 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:29.430393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:29.432074 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:29.432830 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:29.434178 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:29.434685 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:29.434828 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:29.437236 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:29.437248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:29.437253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:29.437286 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:29.440906 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:29.441765 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:29.441770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:29.441774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:29.441930 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:29.442385 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:29.442389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:29.442391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:29.442737 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:29.442981 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:29.443409 (+ 428us) server_negotiation.cc:299] Negotiation successful
0809 05:26:29.443651 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:26:29.445211 21526 catalog_manager.cc:1347] Generated new cluster ID: f63d9f8b07fb4473ad350c17d9678d89
I20260809 05:26:29.445341 21526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:29.458333 21526 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:29.460084 21526 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:29.475828 21526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 390024bd15214eee904a246e3c112144: Generated new TSK 0
I20260809 05:26:29.476794 21526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:29.489960 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21405
I20260809 05:26:29.491408 21484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:29.583771 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21405
2026-08-09T05:26:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:29 dist-test-slave-rmhg krb5kdc[21535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:29 dist-test-slave-rmhg krb5kdc[21535](info): set up 2 sockets
Aug 09 05:26:29 dist-test-slave-rmhg krb5kdc[21535](info): commencing operation
krb5kdc: starting...
W20260809 05:26:30.810564 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:31 dist-test-slave-rmhg krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:31.116554 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:31.461294 21556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:31.461624 21556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:31.461772 21556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:31.469367 21556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:31.469498 21556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:31.469558 21556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:31.469645 21556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:31.469703 21556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:31.480365 21556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:31.481843 21556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:31.483203 21556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:31.495671 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:31.495702 21565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:31.495756 21563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:31.496901 21556 server_base.cc:1034] running on GCE node
I20260809 05:26:31.497794 21556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:31.499259 21556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:31.500515 21556 hybrid_clock.cc:648] HybridClock initialized: now 1786253191500427 us; error 44 us; skew 500 ppm
I20260809 05:26:31.500865 21556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:31 dist-test-slave-rmhg krb5kdc[21535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:31.508275 21556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:31.509996 21556 webserver.cc:460] Webserver started at http://127.0.0.1:38009/ using document root <none> and password file <none>
I20260809 05:26:31.510732 21556 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:31.510855 21556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:31.511174 21556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:31.514391 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "c9a545a5c59f400680256074e30f4831"
format_stamp: "Formatted at 2026-08-09 05:26:31 on dist-test-slave-rmhg"
server_key: "2f6bc07a9768331d2f9ab04800304cb5"
server_key_iv: "c3c4d3237eccecece42587537c133d31"
server_key_version: "encryptionkey@0"
I20260809 05:26:31.515194 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "c9a545a5c59f400680256074e30f4831"
format_stamp: "Formatted at 2026-08-09 05:26:31 on dist-test-slave-rmhg"
server_key: "2f6bc07a9768331d2f9ab04800304cb5"
server_key_iv: "c3c4d3237eccecece42587537c133d31"
server_key_version: "encryptionkey@0"
I20260809 05:26:31.520438 21556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:31.524170 21572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:31.525761 21556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:31.525947 21556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "c9a545a5c59f400680256074e30f4831"
format_stamp: "Formatted at 2026-08-09 05:26:31 on dist-test-slave-rmhg"
server_key: "2f6bc07a9768331d2f9ab04800304cb5"
server_key_iv: "c3c4d3237eccecece42587537c133d31"
server_key_version: "encryptionkey@0"
I20260809 05:26:31.526165 21556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:31.544225 21556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:31.548538 21556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:31.548815 21556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:31.566504 21556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36035
I20260809 05:26:31.566537 21644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36035 every 8 connection(s)
I20260809 05:26:31.567966 21556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:31.568770 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21556
I20260809 05:26:31.568964 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:31.569507 22533 external_mini_cluster.cc:1428] Setting key 0541ea50bd42193705b09a622a1a669f
I20260809 05:26:31.572710 21647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:31.584967 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:31.571420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49629 (local address 127.0.0.1:36035)
0809 05:26:31.572221 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:31.572237 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:31.572288 (+ 51us) server_negotiation.cc:407] Connection header received
0809 05:26:31.574580 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:31.574607 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:31.575120 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:31.575587 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:31.576116 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:31.578165 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:31.579593 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:31.580616 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:31.580794 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:31.581980 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:31.581993 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:31.582013 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:31.582791 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:31.582903 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:31.582937 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:31.583062 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:31.583289 (+ 227us) server_negotiation.cc:299] Negotiation successful
0809 05:26:31.583600 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":336,"threads_started":1}
I20260809 05:26:31.586606 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831: Bootstrap starting.
I20260809 05:26:31.589254 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:31.590220 21647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:31.591254 21647 log.cc:824] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:31.594312 21647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831: No bootstrap required, opened a new log
I20260809 05:26:31.600296 21651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:31.593611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35269 (local address 127.0.0.1:36035)
0809 05:26:31.593787 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:31.593793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:31.594157 (+ 364us) server_negotiation.cc:407] Connection header received
0809 05:26:31.594320 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:31.594325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:31.594406 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:31.594525 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:31.595024 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:31.596075 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:31.597558 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:31.598247 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:31.598379 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:31.599516 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:31.599523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:31.599528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:31.599550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:31.599602 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:31.599623 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:31.599719 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:31.599856 (+ 137us) server_negotiation.cc:299] Negotiation successful
0809 05:26:31.599951 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:26:31.603412 21647 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260809 05:26:31.603826 21647 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:31.603909 21647 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a545a5c59f400680256074e30f4831, State: Initialized, Role: FOLLOWER
I20260809 05:26:31.604547 21647 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260809 05:26:31.604777 21647 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:31.604880 21647 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:31.605032 21647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:31.607703 21647 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260809 05:26:31.608215 21647 leader_election.cc:302] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a545a5c59f400680256074e30f4831; no voters:
I20260809 05:26:31.608758 21647 leader_election.cc:288] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:31.608906 21662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:31.609557 21662 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [term 1 LEADER]: Becoming Leader. State: Replica: c9a545a5c59f400680256074e30f4831, State: Running, Role: LEADER
I20260809 05:26:31.610132 21662 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260809 05:26:31.610877 21647 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:31.612790 21663 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9a545a5c59f400680256074e30f4831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } } }
I20260809 05:26:31.613243 21663 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:31.612823 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9a545a5c59f400680256074e30f4831. Latest consensus state: current_term: 1 leader_uuid: "c9a545a5c59f400680256074e30f4831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a545a5c59f400680256074e30f4831" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } } }
I20260809 05:26:31.617612 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:31.617842 21670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:31.623066 21670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:31.632107 21670 catalog_manager.cc:1347] Generated new cluster ID: c40c34d4ad5b4166ad56d26384694c2b
I20260809 05:26:31.632231 21670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:31.644577 21670 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:31.646121 21670 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:31.652618 21670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9a545a5c59f400680256074e30f4831: Generated new TSK 0
I20260809 05:26:31.653525 21670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:31.662299 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21556
W20260809 05:26:31.663661 21639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49629
W20260809 05:26:31.664373 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:31.664520 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21556
2026-08-09T05:26:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:31 dist-test-slave-rmhg krb5kdc[21693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:31 dist-test-slave-rmhg krb5kdc[21693](info): set up 2 sockets
Aug 09 05:26:31 dist-test-slave-rmhg krb5kdc[21693](info): commencing operation
krb5kdc: starting...
W20260809 05:26:32.916139 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:33 dist-test-slave-rmhg krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:33.232803 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:33.571630 21718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:33.571949 21718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:33.572069 21718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:33.579488 21718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:33.579612 21718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:33.579664 21718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:33.579747 21718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:33.579806 21718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:33.589954 21718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:33.591331 21718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:33.592691 21718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:33.604367 21725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:33.604328 21729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:33.604411 21726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:33.605412 21718 server_base.cc:1034] running on GCE node
I20260809 05:26:33.606470 21718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:33.608016 21718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:33.609274 21718 hybrid_clock.cc:648] HybridClock initialized: now 1786253193609214 us; error 40 us; skew 500 ppm
I20260809 05:26:33.609656 21718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:33 dist-test-slave-rmhg krb5kdc[21693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:33.612689 21718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:33.614216 21718 webserver.cc:460] Webserver started at http://127.0.0.1:36193/ using document root <none> and password file <none>
I20260809 05:26:33.614890 21718 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:33.615023 21718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:33.615330 21718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:33.618403 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "c585472e3592403194ca1476d880b3f9"
format_stamp: "Formatted at 2026-08-09 05:26:33 on dist-test-slave-rmhg"
server_key: "43f6949d7bdbeb167e75eace6ea5f18e"
server_key_iv: "f9615ab343f5e9fec1ab00a4772ccd30"
server_key_version: "encryptionkey@0"
I20260809 05:26:33.619163 21718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "c585472e3592403194ca1476d880b3f9"
format_stamp: "Formatted at 2026-08-09 05:26:33 on dist-test-slave-rmhg"
server_key: "43f6949d7bdbeb167e75eace6ea5f18e"
server_key_iv: "f9615ab343f5e9fec1ab00a4772ccd30"
server_key_version: "encryptionkey@0"
I20260809 05:26:33.624305 21718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:26:33.627782 21738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:33.629060 21718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:33.629256 21718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "c585472e3592403194ca1476d880b3f9"
format_stamp: "Formatted at 2026-08-09 05:26:33 on dist-test-slave-rmhg"
server_key: "43f6949d7bdbeb167e75eace6ea5f18e"
server_key_iv: "f9615ab343f5e9fec1ab00a4772ccd30"
server_key_version: "encryptionkey@0"
I20260809 05:26:33.629442 21718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:33.635138 21718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:33.639144 21718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:33.639430 21718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:33.657599 21718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34079
I20260809 05:26:33.657652 21805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34079 every 8 connection(s)
I20260809 05:26:33.659018 21718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:33.662707 21807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:33.664527 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21718
I20260809 05:26:33.664695 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:33.665138 22533 external_mini_cluster.cc:1428] Setting key 69dcbeb751f1c13c545fc0e4448fdba4
I20260809 05:26:33.674373 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:33.667101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44853 (local address 127.0.0.1:34079)
0809 05:26:33.667964 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:33.667981 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:33.668038 (+ 57us) server_negotiation.cc:407] Connection header received
0809 05:26:33.670348 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:33.670379 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:33.670832 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:33.671267 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:33.672269 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:33.672281 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:33.672298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:33.672319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:33.672389 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:33.672424 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:33.672667 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:33.672802 (+ 135us) server_negotiation.cc:299] Negotiation successful
0809 05:26:33.673012 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":356,"threads_started":1}
I20260809 05:26:33.676312 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9: Bootstrap starting.
I20260809 05:26:33.678516 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:33.679868 21807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:33.680924 21807 log.cc:824] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:33.684857 21807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9: No bootstrap required, opened a new log
I20260809 05:26:33.686177 21810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:33.683163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49041 (local address 127.0.0.1:34079)
0809 05:26:33.683339 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:33.683344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:33.684101 (+ 757us) server_negotiation.cc:407] Connection header received
0809 05:26:33.684299 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:33.684303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:33.684354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:33.684457 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:33.685517 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:33.685522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:33.685525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:33.685540 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:33.685585 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:33.685600 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:33.685692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:33.685843 (+ 151us) server_negotiation.cc:299] Negotiation successful
0809 05:26:33.685894 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:26:33.693003 21807 raft_consensus.cc:348] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260809 05:26:33.693471 21807 raft_consensus.cc:374] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:33.693581 21807 raft_consensus.cc:729] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c585472e3592403194ca1476d880b3f9, State: Initialized, Role: FOLLOWER
I20260809 05:26:33.694164 21807 consensus_queue.cc:260] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260809 05:26:33.694392 21807 raft_consensus.cc:388] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:33.694543 21807 raft_consensus.cc:482] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:33.694725 21807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:33.696903 21807 raft_consensus.cc:504] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260809 05:26:33.697427 21807 leader_election.cc:302] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c585472e3592403194ca1476d880b3f9; no voters:
I20260809 05:26:33.698055 21807 leader_election.cc:288] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:33.698206 21818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:33.698834 21818 raft_consensus.cc:686] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [term 1 LEADER]: Becoming Leader. State: Replica: c585472e3592403194ca1476d880b3f9, State: Running, Role: LEADER
I20260809 05:26:33.699438 21818 consensus_queue.cc:237] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } }
I20260809 05:26:33.700268 21807 sys_catalog.cc:564] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:33.702433 21819 sys_catalog.cc:455] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c585472e3592403194ca1476d880b3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } } }
I20260809 05:26:33.702878 21819 sys_catalog.cc:458] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:33.703076 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c585472e3592403194ca1476d880b3f9. Latest consensus state: current_term: 1 leader_uuid: "c585472e3592403194ca1476d880b3f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c585472e3592403194ca1476d880b3f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34079 } } }
I20260809 05:26:33.703400 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:33.704872 21831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:33.709601 21831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:33.717408 21831 catalog_manager.cc:1347] Generated new cluster ID: d8b4206e31064e58aa4afe03674dc849
I20260809 05:26:33.717538 21831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:33.726799 21831 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:33.727283 21831 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:33.739130 21831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c585472e3592403194ca1476d880b3f9: Generated new TSK 0
I20260809 05:26:33.740023 21831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:33.752215 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21718
W20260809 05:26:33.753500 21801 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44853
W20260809 05:26:33.754176 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:33.754315 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21718
2026-08-09T05:26:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2085 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:33 dist-test-slave-rmhg krb5kdc[21849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:33 dist-test-slave-rmhg krb5kdc[21849](info): set up 2 sockets
Aug 09 05:26:33 dist-test-slave-rmhg krb5kdc[21849](info): commencing operation
krb5kdc: starting...
W20260809 05:26:34.991853 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:35 dist-test-slave-rmhg krb5kdc[21849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:35.310580 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:35.654618 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:35.654937 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:35.655036 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:35.662479 21866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:35.662611 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:35.662657 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:35.662720 21866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:35.662763 21866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:35.672969 21866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:35.674295 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:35.675593 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:35.687278 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:35.687414 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:35.687403 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:35.688339 21866 server_base.cc:1034] running on GCE node
I20260809 05:26:35.689229 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:35.690776 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:35.691985 21866 hybrid_clock.cc:648] HybridClock initialized: now 1786253195691939 us; error 40 us; skew 500 ppm
I20260809 05:26:35.692332 21866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:35 dist-test-slave-rmhg krb5kdc[21849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:35.695456 21866 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:35.696992 21866 webserver.cc:460] Webserver started at http://127.0.0.1:38579/ using document root <none> and password file <none>
I20260809 05:26:35.697729 21866 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:35.697839 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:35.698154 21866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:35.701184 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "e8c1552583964e2c9d0dfce9a5db22e7"
format_stamp: "Formatted at 2026-08-09 05:26:35 on dist-test-slave-rmhg"
server_key: "0796afe6fabee37dafe08bb8cb280468"
server_key_iv: "57efd3c9cc5c49efc63e157988068fa3"
server_key_version: "encryptionkey@0"
I20260809 05:26:35.702025 21866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "e8c1552583964e2c9d0dfce9a5db22e7"
format_stamp: "Formatted at 2026-08-09 05:26:35 on dist-test-slave-rmhg"
server_key: "0796afe6fabee37dafe08bb8cb280468"
server_key_iv: "57efd3c9cc5c49efc63e157988068fa3"
server_key_version: "encryptionkey@0"
I20260809 05:26:35.707329 21866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260809 05:26:35.710860 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:35.712231 21866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:26:35.712389 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "e8c1552583964e2c9d0dfce9a5db22e7"
format_stamp: "Formatted at 2026-08-09 05:26:35 on dist-test-slave-rmhg"
server_key: "0796afe6fabee37dafe08bb8cb280468"
server_key_iv: "57efd3c9cc5c49efc63e157988068fa3"
server_key_version: "encryptionkey@0"
I20260809 05:26:35.712568 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:35.725785 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:35.730221 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:35.730520 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:35.748174 21866 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37271
I20260809 05:26:35.748184 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37271 every 8 connection(s)
I20260809 05:26:35.749647 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:35.752660 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 21866
I20260809 05:26:35.752893 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:35.753441 22533 external_mini_cluster.cc:1428] Setting key 2dbc85ccd094c95785caa192e1022e42
I20260809 05:26:35.754570 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:35.766402 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7: Bootstrap starting.
Aug 09 05:26:35 dist-test-slave-rmhg krb5kdc[21849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 05:26:35.769922 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:35.771090 21964 log.cc:824] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:35.773955 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7: No bootstrap required, opened a new log
I20260809 05:26:35.775739 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:35.755266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:58983 (local address 127.0.0.1:37271)
0809 05:26:35.756023 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:35.756035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:35.756076 (+ 41us) server_negotiation.cc:407] Connection header received
0809 05:26:35.757767 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:35.757791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:35.758210 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:35.758619 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:35.760091 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:35.761477 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:35.762785 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:35.763506 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:35.763697 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:35.768080 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:35.768093 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:35.768109 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:35.768154 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:35.771372 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:35.772240 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:35.772248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:35.772254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:35.772427 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:35.772903 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:35.772907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:35.772909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:35.773503 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:35.773818 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:35.774344 (+ 526us) server_negotiation.cc:299] Negotiation successful
0809 05:26:35.774681 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":324,"threads_started":1}
I20260809 05:26:35.780251 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:35.782732 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260809 05:26:35.783165 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:35.783264 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c1552583964e2c9d0dfce9a5db22e7, State: Initialized, Role: FOLLOWER
I20260809 05:26:35.783849 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260809 05:26:35.784060 21964 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:35.784190 21964 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:35.784346 21964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:35.786943 21964 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260809 05:26:35.787552 21964 leader_election.cc:302] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8c1552583964e2c9d0dfce9a5db22e7; no voters:
I20260809 05:26:35.788184 21964 leader_election.cc:288] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:35.788285 21980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:35.789717 21980 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [term 1 LEADER]: Becoming Leader. State: Replica: e8c1552583964e2c9d0dfce9a5db22e7, State: Running, Role: LEADER
I20260809 05:26:35.790386 21980 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } }
I20260809 05:26:35.790692 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:35.798910 21982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8c1552583964e2c9d0dfce9a5db22e7. Latest consensus state: current_term: 1 leader_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260809 05:26:35.799583 21982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:35.799183 21981 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8c1552583964e2c9d0dfce9a5db22e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37271 } } }
I20260809 05:26:35.800227 21981 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:35.802453 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:35.786812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:51853 (local address 127.0.0.1:37271)
0809 05:26:35.786982 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:35.786989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:35.787426 (+ 437us) server_negotiation.cc:407] Connection header received
0809 05:26:35.787615 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:35.787620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:35.787692 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:35.787783 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:35.789534 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:35.790574 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:35.791977 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:35.792495 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:35.792615 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:35.795231 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:35.795237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:35.795241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:35.795273 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:35.799342 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:35.800353 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:35.800359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:35.800361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:35.800514 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:35.800980 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:35.800984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:35.800986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:35.801364 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:35.801657 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:35.801981 (+ 324us) server_negotiation.cc:299] Negotiation successful
0809 05:26:35.802157 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260809 05:26:35.810760 21988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:35.815542 21988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:35.823709 21988 catalog_manager.cc:1347] Generated new cluster ID: 23b3cbec514a4d819292684c8e015dfa
I20260809 05:26:35.823827 21988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:35.846133 21988 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:35.847555 21988 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:35.857909 21988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8c1552583964e2c9d0dfce9a5db22e7: Generated new TSK 0
I20260809 05:26:35.858994 21988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:35.868894 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 21866
I20260809 05:26:35.870908 21958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:35.962256 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 21866
W20260809 05:26:35.986361 21847 connection.cc:535] client connection to 127.0.0.1:37271 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37271): Connection reset by peer (error 104)
2026-08-09T05:26:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:36 dist-test-slave-rmhg krb5kdc[22014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:36 dist-test-slave-rmhg krb5kdc[22014](info): set up 2 sockets
Aug 09 05:26:36 dist-test-slave-rmhg krb5kdc[22014](info): commencing operation
krb5kdc: starting...
W20260809 05:26:37.186949 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.008s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:37 dist-test-slave-rmhg krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:37.509577 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:37.854004 22030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:37.854314 22030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:37.854449 22030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:37.861711 22030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:37.861829 22030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:37.861903 22030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:37.862010 22030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:37.862066 22030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:37.871963 22030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:37.873333 22030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:37.874668 22030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:37.886363 22041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:37.886325 22045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:37.886273 22042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:37.887174 22030 server_base.cc:1034] running on GCE node
I20260809 05:26:37.888055 22030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:37.889650 22030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:37.890862 22030 hybrid_clock.cc:648] HybridClock initialized: now 1786253197890812 us; error 46 us; skew 500 ppm
I20260809 05:26:37.891237 22030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:37 dist-test-slave-rmhg krb5kdc[22014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:37.897733 22030 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:37.899292 22030 webserver.cc:460] Webserver started at http://127.0.0.1:36249/ using document root <none> and password file <none>
I20260809 05:26:37.899971 22030 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:37.900102 22030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:37.900447 22030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:37.903522 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9"
format_stamp: "Formatted at 2026-08-09 05:26:37 on dist-test-slave-rmhg"
server_key: "374a35f92d7aba03240d72a851b7e723"
server_key_iv: "44a543a8695ea38bf45304e36ec2d79d"
server_key_version: "encryptionkey@0"
I20260809 05:26:37.904275 22030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9"
format_stamp: "Formatted at 2026-08-09 05:26:37 on dist-test-slave-rmhg"
server_key: "374a35f92d7aba03240d72a851b7e723"
server_key_iv: "44a543a8695ea38bf45304e36ec2d79d"
server_key_version: "encryptionkey@0"
I20260809 05:26:37.909394 22030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:26:37.912825 22054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:37.914157 22030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:37.914310 22030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9"
format_stamp: "Formatted at 2026-08-09 05:26:37 on dist-test-slave-rmhg"
server_key: "374a35f92d7aba03240d72a851b7e723"
server_key_iv: "44a543a8695ea38bf45304e36ec2d79d"
server_key_version: "encryptionkey@0"
I20260809 05:26:37.914482 22030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:37.940241 22030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:37.944612 22030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:37.944890 22030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:37.961822 22030 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44181
I20260809 05:26:37.961856 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44181 every 8 connection(s)
I20260809 05:26:37.963210 22030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:37.967080 22136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:37.971854 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22030
I20260809 05:26:37.972059 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:37.972630 22533 external_mini_cluster.cc:1428] Setting key 1d601fd3075090290e2758827b9dcd09
I20260809 05:26:37.978924 22136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9: Bootstrap starting.
I20260809 05:26:37.982517 22136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:37.983791 22136 log.cc:824] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:37.987361 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:37.974584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:38763 (local address 127.0.0.1:44181)
0809 05:26:37.975335 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:37.975352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:37.975398 (+ 46us) server_negotiation.cc:407] Connection header received
0809 05:26:37.977368 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:37.977392 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:37.977802 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:37.978127 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:37.978732 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:37.980108 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:37.982003 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:37.982752 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:37.982918 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:37.984217 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:37.984225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:37.984240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:37.985340 (+ 1100us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:37.985442 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:37.985476 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:37.985664 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:37.985895 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 05:26:37.986095 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":326,"threads_started":1}
I20260809 05:26:37.987977 22136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9: No bootstrap required, opened a new log
I20260809 05:26:37.991137 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:37.995949 22136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260809 05:26:37.996346 22136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:37.996455 22136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7b4a5b565a4c3a9bd725ba8e74bbd9, State: Initialized, Role: FOLLOWER
I20260809 05:26:37.997051 22136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260809 05:26:37.997339 22136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:37.997488 22136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:37.997666 22136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:37.999912 22136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260809 05:26:38.000523 22136 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7b4a5b565a4c3a9bd725ba8e74bbd9; no voters:
I20260809 05:26:38.001247 22136 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:38.001372 22151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:38.001905 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:37.995380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:38799 (local address 127.0.0.1:44181)
0809 05:26:37.995552 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:37.995558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:37.995983 (+ 425us) server_negotiation.cc:407] Connection header received
0809 05:26:37.996168 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:37.996172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:37.996236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:37.996314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:37.996859 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:37.997784 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:37.999087 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:37.999551 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:37.999667 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:38.000860 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:38.000866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:38.000869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:38.000886 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:38.000995 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:38.001019 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:38.001171 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:38.001476 (+ 305us) server_negotiation.cc:299] Negotiation successful
0809 05:26:38.001531 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260809 05:26:38.002987 22151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7b4a5b565a4c3a9bd725ba8e74bbd9, State: Running, Role: LEADER
I20260809 05:26:38.003578 22151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } }
I20260809 05:26:38.003774 22136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:38.006284 22152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } } }
I20260809 05:26:38.006603 22152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:38.007838 22159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:38.013244 22153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7b4a5b565a4c3a9bd725ba8e74bbd9. Latest consensus state: current_term: 1 leader_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7b4a5b565a4c3a9bd725ba8e74bbd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44181 } } }
I20260809 05:26:38.013525 22153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:38.017364 22159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:38.026468 22159 catalog_manager.cc:1347] Generated new cluster ID: 5d08c429028e4491ac1d44c40e7e52fd
I20260809 05:26:38.026595 22159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:38.036087 22159 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:38.037987 22159 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:38.051733 22159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b7b4a5b565a4c3a9bd725ba8e74bbd9: Generated new TSK 0
I20260809 05:26:38.052979 22159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:38.066855 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22030
W20260809 05:26:38.068332 22129 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:38763
W20260809 05:26:38.069124 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22030: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:38.069314 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22030
W20260809 05:26:38.097438 22009 connection.cc:535] client connection to 127.0.0.1:44181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44181): Connection reset by peer (error 104)
2026-08-09T05:26:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2111 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:38 dist-test-slave-rmhg krb5kdc[22179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:38 dist-test-slave-rmhg krb5kdc[22179](info): set up 2 sockets
Aug 09 05:26:38 dist-test-slave-rmhg krb5kdc[22179](info): commencing operation
krb5kdc: starting...
W20260809 05:26:39.310904 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:39 dist-test-slave-rmhg krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:39.630631 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:39.974808 22199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:39.975167 22199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:39.975310 22199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:39.982730 22199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:39.982877 22199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:39.982961 22199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:39.983047 22199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:39.983109 22199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:39.993721 22199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:39.995158 22199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:39.996721 22199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:40.008764 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:40.008831 22208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:40.008795 22210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:40.009747 22199 server_base.cc:1034] running on GCE node
I20260809 05:26:40.010623 22199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:40.012312 22199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:40.013556 22199 hybrid_clock.cc:648] HybridClock initialized: now 1786253200013488 us; error 60 us; skew 500 ppm
I20260809 05:26:40.013921 22199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:40 dist-test-slave-rmhg krb5kdc[22179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:40.017373 22199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:40.018997 22199 webserver.cc:460] Webserver started at http://127.0.0.1:41827/ using document root <none> and password file <none>
I20260809 05:26:40.019773 22199 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:40.019903 22199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:40.020220 22199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:40.023415 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "008b212418ee4d689a1e4ff7b434c5cc"
format_stamp: "Formatted at 2026-08-09 05:26:40 on dist-test-slave-rmhg"
server_key: "9d0e3e6541d795ad75a816333c590e0f"
server_key_iv: "1bcd30c8e2027f5e74f07c994b996ce5"
server_key_version: "encryptionkey@0"
I20260809 05:26:40.024205 22199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "008b212418ee4d689a1e4ff7b434c5cc"
format_stamp: "Formatted at 2026-08-09 05:26:40 on dist-test-slave-rmhg"
server_key: "9d0e3e6541d795ad75a816333c590e0f"
server_key_iv: "1bcd30c8e2027f5e74f07c994b996ce5"
server_key_version: "encryptionkey@0"
I20260809 05:26:40.029695 22199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260809 05:26:40.033440 22219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:40.034838 22199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:40.035039 22199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "008b212418ee4d689a1e4ff7b434c5cc"
format_stamp: "Formatted at 2026-08-09 05:26:40 on dist-test-slave-rmhg"
server_key: "9d0e3e6541d795ad75a816333c590e0f"
server_key_iv: "1bcd30c8e2027f5e74f07c994b996ce5"
server_key_version: "encryptionkey@0"
I20260809 05:26:40.035228 22199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:40.040982 22199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:40.045281 22199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:40.045555 22199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:40.062923 22199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35521
I20260809 05:26:40.062964 22291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35521 every 8 connection(s)
I20260809 05:26:40.064352 22199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:40.068080 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:40.072345 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22199
I20260809 05:26:40.072551 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:40.073011 22533 external_mini_cluster.cc:1428] Setting key b724144f6bfdbf875f823c1916732425
I20260809 05:26:40.079102 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc: Bootstrap starting.
I20260809 05:26:40.080847 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:40.074889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:50181 (local address 127.0.0.1:35521)
0809 05:26:40.075724 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:40.075739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:40.075782 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:26:40.077570 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:40.077596 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:40.077938 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:40.078250 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:40.079330 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:40.079336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:40.079349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:40.079364 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:40.079423 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:40.079442 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:40.079637 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:40.079755 (+ 118us) server_negotiation.cc:299] Negotiation successful
0809 05:26:40.079923 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":344,"threads_started":1}
I20260809 05:26:40.082369 22292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:40.083494 22292 log.cc:824] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:40.084328 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:40.086262 22292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc: No bootstrap required, opened a new log
I20260809 05:26:40.091113 22296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:40.088683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:48347 (local address 127.0.0.1:35521)
0809 05:26:40.088833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:40.088837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:40.089334 (+ 497us) server_negotiation.cc:407] Connection header received
0809 05:26:40.089476 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:40.089482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:40.089540 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:40.089648 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:40.090464 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:40.090477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:40.090482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:40.090500 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:40.090555 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:40.090574 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:40.090701 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:40.090741 (+ 40us) server_negotiation.cc:299] Negotiation successful
0809 05:26:40.090801 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260809 05:26:40.094601 22292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260809 05:26:40.094949 22292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:40.095036 22292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008b212418ee4d689a1e4ff7b434c5cc, State: Initialized, Role: FOLLOWER
I20260809 05:26:40.095546 22292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260809 05:26:40.095757 22292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:40.095852 22292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:40.095996 22292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:40.098289 22292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260809 05:26:40.098814 22292 leader_election.cc:302] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 008b212418ee4d689a1e4ff7b434c5cc; no voters:
I20260809 05:26:40.099370 22292 leader_election.cc:288] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:40.099498 22303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:40.100159 22303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [term 1 LEADER]: Becoming Leader. State: Replica: 008b212418ee4d689a1e4ff7b434c5cc, State: Running, Role: LEADER
I20260809 05:26:40.100739 22303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260809 05:26:40.101418 22292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:40.103646 22304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "008b212418ee4d689a1e4ff7b434c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260809 05:26:40.104519 22304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:40.104197 22305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 008b212418ee4d689a1e4ff7b434c5cc. Latest consensus state: current_term: 1 leader_uuid: "008b212418ee4d689a1e4ff7b434c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008b212418ee4d689a1e4ff7b434c5cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260809 05:26:40.104918 22305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:40.106561 22310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:40.111672 22310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:40.121176 22310 catalog_manager.cc:1347] Generated new cluster ID: f8f8fd24db2a42118363394df91231ed
I20260809 05:26:40.121307 22310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:40.145473 22310 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:40.146044 22310 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:40.153853 22310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 008b212418ee4d689a1e4ff7b434c5cc: Generated new TSK 0
I20260809 05:26:40.154668 22310 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:40.160882 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22199
W20260809 05:26:40.162039 22285 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:50181
W20260809 05:26:40.162782 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22199: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:40.162931 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22199
2026-08-09T05:26:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2090 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:40 dist-test-slave-rmhg krb5kdc[22329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:40 dist-test-slave-rmhg krb5kdc[22329](info): set up 2 sockets
Aug 09 05:26:40 dist-test-slave-rmhg krb5kdc[22329](info): commencing operation
krb5kdc: starting...
W20260809 05:26:41.394527 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:41 dist-test-slave-rmhg krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:41.715915 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:42.057245 22351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:42.057569 22351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:42.057704 22351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:42.065209 22351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:42.065340 22351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:42.065395 22351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:42.065488 22351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:42.065544 22351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:42.075503 22351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:42.076849 22351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:42.078191 22351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:42.089668 22358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:42.089641 22359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:42.089704 22361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:42.090520 22351 server_base.cc:1034] running on GCE node
I20260809 05:26:42.091262 22351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:42.092679 22351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:42.093887 22351 hybrid_clock.cc:648] HybridClock initialized: now 1786253202093837 us; error 41 us; skew 500 ppm
I20260809 05:26:42.094234 22351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:42 dist-test-slave-rmhg krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:42.097440 22351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:42.099076 22351 webserver.cc:460] Webserver started at http://127.0.0.1:45819/ using document root <none> and password file <none>
I20260809 05:26:42.099798 22351 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:42.099927 22351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:42.100246 22351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:42.103667 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "0be4ebf9818d48adadc2824c48ed849f"
format_stamp: "Formatted at 2026-08-09 05:26:42 on dist-test-slave-rmhg"
server_key: "cfb3de6c9120cd728b54c37a9d868ca3"
server_key_iv: "7cc934f19504377c2f86286377138537"
server_key_version: "encryptionkey@0"
I20260809 05:26:42.104774 22351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "0be4ebf9818d48adadc2824c48ed849f"
format_stamp: "Formatted at 2026-08-09 05:26:42 on dist-test-slave-rmhg"
server_key: "cfb3de6c9120cd728b54c37a9d868ca3"
server_key_iv: "7cc934f19504377c2f86286377138537"
server_key_version: "encryptionkey@0"
I20260809 05:26:42.113663 22351 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260809 05:26:42.117676 22369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:42.118980 22351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:26:42.119139 22351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "0be4ebf9818d48adadc2824c48ed849f"
format_stamp: "Formatted at 2026-08-09 05:26:42 on dist-test-slave-rmhg"
server_key: "cfb3de6c9120cd728b54c37a9d868ca3"
server_key_iv: "7cc934f19504377c2f86286377138537"
server_key_version: "encryptionkey@0"
I20260809 05:26:42.119329 22351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:42.131796 22351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:42.136090 22351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:42.136376 22351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:42.154271 22351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46521
I20260809 05:26:42.154282 22437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46521 every 8 connection(s)
I20260809 05:26:42.155673 22351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:42.157640 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22351
I20260809 05:26:42.157836 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:42.158407 22533 external_mini_cluster.cc:1428] Setting key e599f446bb0ae758a17ee950b7aca689
I20260809 05:26:42.160240 22439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:42.173460 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f: Bootstrap starting.
Aug 09 05:26:42 dist-test-slave-rmhg krb5kdc[22329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260809 05:26:42.177443 22439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:42.178948 22439 log.cc:824] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:42.182672 22439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f: No bootstrap required, opened a new log
I20260809 05:26:42.183892 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:42.160282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:59661 (local address 127.0.0.1:46521)
0809 05:26:42.161046 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:42.161062 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:42.161114 (+ 52us) server_negotiation.cc:407] Connection header received
0809 05:26:42.163266 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:42.163295 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:42.163843 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:42.164293 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:42.165811 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:42.167895 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:42.169226 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:42.170282 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:42.170492 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:42.174641 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:42.174652 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:42.174673 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:42.174721 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:42.178765 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:42.179812 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:42.179819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:42.179826 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:42.180018 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:42.180596 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:42.180601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:42.180604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:42.181337 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:42.181712 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:42.182174 (+ 462us) server_negotiation.cc:299] Negotiation successful
0809 05:26:42.182536 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":294,"threads_started":1}
I20260809 05:26:42.187979 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:42.192184 22439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } }
I20260809 05:26:42.192651 22439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:42.192764 22439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0be4ebf9818d48adadc2824c48ed849f, State: Initialized, Role: FOLLOWER
I20260809 05:26:42.193454 22439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } }
I20260809 05:26:42.193743 22439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:42.193895 22439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:42.194083 22439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:42.197027 22439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } }
I20260809 05:26:42.197626 22439 leader_election.cc:302] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0be4ebf9818d48adadc2824c48ed849f; no voters:
I20260809 05:26:42.198313 22439 leader_election.cc:288] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:42.198419 22453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:42.200778 22439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:42.205477 22453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [term 1 LEADER]: Becoming Leader. State: Replica: 0be4ebf9818d48adadc2824c48ed849f, State: Running, Role: LEADER
I20260809 05:26:42.206156 22453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } }
I20260809 05:26:42.209055 22454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0be4ebf9818d48adadc2824c48ed849f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } } }
I20260809 05:26:42.209568 22454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:42.209971 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0be4ebf9818d48adadc2824c48ed849f. Latest consensus state: current_term: 1 leader_uuid: "0be4ebf9818d48adadc2824c48ed849f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0be4ebf9818d48adadc2824c48ed849f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46521 } } }
I20260809 05:26:42.210598 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:42.211051 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:42.193389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:59577 (local address 127.0.0.1:46521)
0809 05:26:42.193579 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:42.193585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:42.194110 (+ 525us) server_negotiation.cc:407] Connection header received
0809 05:26:42.194316 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:42.194319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:42.194390 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:42.194476 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:42.196307 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:42.197081 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:42.198818 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:42.199561 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:42.199683 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:42.202957 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:42.202965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:42.202971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:42.203017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:42.206922 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:42.208061 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:42.208067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:42.208070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:42.208268 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:42.208905 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:42.208911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:42.208915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:42.209402 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:42.209705 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:42.210470 (+ 765us) server_negotiation.cc:299] Negotiation successful
0809 05:26:42.210677 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:26:42.214496 22463 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:42.220470 22463 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:42.230367 22463 catalog_manager.cc:1347] Generated new cluster ID: 3add44364dd24922a84bdab020d3712f
I20260809 05:26:42.230571 22463 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:42.248633 22463 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:42.250571 22463 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:42.260588 22463 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0be4ebf9818d48adadc2824c48ed849f: Generated new TSK 0
I20260809 05:26:42.261608 22463 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:42.273631 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22351
I20260809 05:26:42.274923 22433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:42.365507 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22351
W20260809 05:26:42.389858 22326 connection.cc:535] client connection to 127.0.0.1:46521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46521): Connection reset by peer (error 104)
2026-08-09T05:26:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:42 dist-test-slave-rmhg krb5kdc[22482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:42 dist-test-slave-rmhg krb5kdc[22482](info): set up 2 sockets
Aug 09 05:26:42 dist-test-slave-rmhg krb5kdc[22482](info): commencing operation
krb5kdc: starting...
W20260809 05:26:43.591066 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.003s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:43 dist-test-slave-rmhg krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:43.918347 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:44.257321 22502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:44.257632 22502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:44.257761 22502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:44.265045 22502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:44.265218 22502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:44.265298 22502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:44.265380 22502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:44.265436 22502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:44.276575 22502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:44.278021 22502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:44.279327 22502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:44.291031 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:44.291051 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:44.290989 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:44.292084 22502 server_base.cc:1034] running on GCE node
I20260809 05:26:44.292896 22502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:44.294373 22502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:44.295604 22502 hybrid_clock.cc:648] HybridClock initialized: now 1786253204295543 us; error 45 us; skew 500 ppm
I20260809 05:26:44.296032 22502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:44 dist-test-slave-rmhg krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:44.303476 22502 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:44.305279 22502 webserver.cc:460] Webserver started at http://127.0.0.1:35189/ using document root <none> and password file <none>
I20260809 05:26:44.306059 22502 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:44.306200 22502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:44.306533 22502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:44.309810 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "84216101b2e04babb69907088f6b2796"
format_stamp: "Formatted at 2026-08-09 05:26:44 on dist-test-slave-rmhg"
server_key: "ec2e515d002ee91a188e09ab30c34755"
server_key_iv: "3bca8bdc212b8fa15ae2950f16720aa9"
server_key_version: "encryptionkey@0"
I20260809 05:26:44.310616 22502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "84216101b2e04babb69907088f6b2796"
format_stamp: "Formatted at 2026-08-09 05:26:44 on dist-test-slave-rmhg"
server_key: "ec2e515d002ee91a188e09ab30c34755"
server_key_iv: "3bca8bdc212b8fa15ae2950f16720aa9"
server_key_version: "encryptionkey@0"
I20260809 05:26:44.315915 22502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260809 05:26:44.319481 22529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:44.320881 22502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260809 05:26:44.321053 22502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "84216101b2e04babb69907088f6b2796"
format_stamp: "Formatted at 2026-08-09 05:26:44 on dist-test-slave-rmhg"
server_key: "ec2e515d002ee91a188e09ab30c34755"
server_key_iv: "3bca8bdc212b8fa15ae2950f16720aa9"
server_key_version: "encryptionkey@0"
I20260809 05:26:44.321283 22502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:44.336272 22502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:44.341017 22502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:44.341353 22502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:44.359279 22502 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35825
I20260809 05:26:44.359313 22610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35825 every 8 connection(s)
I20260809 05:26:44.360651 22502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:44.364280 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:44.368562 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22502
I20260809 05:26:44.368762 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:44.369314 22533 external_mini_cluster.cc:1428] Setting key c6047b772a04c33032a423811ae96d7f
I20260809 05:26:44.375304 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796: Bootstrap starting.
I20260809 05:26:44.379565 22611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:44.380983 22611 log.cc:824] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:44.385392 22611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796: No bootstrap required, opened a new log
I20260809 05:26:44.386332 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:44.371221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:38513 (local address 127.0.0.1:35825)
0809 05:26:44.371940 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:44.371961 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:44.372004 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:26:44.374092 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:44.374118 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:44.374529 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:44.374885 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:44.375447 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:44.377451 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:44.379367 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:44.380491 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:44.380721 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:44.382798 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:44.382811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:44.382835 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:44.383642 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:44.383761 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:44.383800 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:44.383978 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:44.384661 (+ 683us) server_negotiation.cc:299] Negotiation successful
0809 05:26:44.384864 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":289,"threads_started":1}
I20260809 05:26:44.395053 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:44.396353 22611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } }
I20260809 05:26:44.396847 22611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:44.396946 22611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84216101b2e04babb69907088f6b2796, State: Initialized, Role: FOLLOWER
I20260809 05:26:44.397650 22611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } }
I20260809 05:26:44.397945 22611 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:44.398084 22611 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:44.398264 22611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:44.401528 22611 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } }
I20260809 05:26:44.402184 22611 leader_election.cc:302] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84216101b2e04babb69907088f6b2796; no voters:
I20260809 05:26:44.402812 22611 leader_election.cc:288] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:44.402976 22621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:44.404872 22611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:44.405408 22621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [term 1 LEADER]: Becoming Leader. State: Replica: 84216101b2e04babb69907088f6b2796, State: Running, Role: LEADER
I20260809 05:26:44.405972 22621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } }
I20260809 05:26:44.408676 22614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:44.400418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:33105 (local address 127.0.0.1:35825)
0809 05:26:44.400626 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:44.400633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:44.401084 (+ 451us) server_negotiation.cc:407] Connection header received
0809 05:26:44.401319 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:44.401322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:44.401394 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:44.401516 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:44.402066 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:44.403064 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:44.405174 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:44.405948 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:44.406096 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:44.407440 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:44.407446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:44.407462 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:44.407489 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:44.407541 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:44.407563 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:44.407694 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:44.408209 (+ 515us) server_negotiation.cc:299] Negotiation successful
0809 05:26:44.408276 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:44.410506 22623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84216101b2e04babb69907088f6b2796. Latest consensus state: current_term: 1 leader_uuid: "84216101b2e04babb69907088f6b2796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } } }
I20260809 05:26:44.410883 22623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:44.411412 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84216101b2e04babb69907088f6b2796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84216101b2e04babb69907088f6b2796" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35825 } } }
I20260809 05:26:44.412346 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:44.413335 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:44.418632 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:44.427631 22637 catalog_manager.cc:1347] Generated new cluster ID: a8893b224c144ed68fe6a1e2fdba15ad
I20260809 05:26:44.427763 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:44.441848 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:44.443154 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:44.451607 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84216101b2e04babb69907088f6b2796: Generated new TSK 0
I20260809 05:26:44.452562 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:44.464246 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22502
W20260809 05:26:44.466002 22601 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:38513
W20260809 05:26:44.466797 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22502: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:44.466931 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22502
W20260809 05:26:44.494692 22479 connection.cc:535] client connection to 127.0.0.1:35825 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35825): Connection reset by peer (error 104)
2026-08-09T05:26:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2104 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:44 dist-test-slave-rmhg krb5kdc[22653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:44 dist-test-slave-rmhg krb5kdc[22653](info): set up 2 sockets
Aug 09 05:26:44 dist-test-slave-rmhg krb5kdc[22653](info): commencing operation
krb5kdc: starting...
W20260809 05:26:45.692198 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:45 dist-test-slave-rmhg krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260809 05:26:46.006598 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:46.349203 22672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:46.349586 22672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:46.349725 22672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:46.357127 22672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:46.357280 22672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:46.357359 22672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:46.357435 22672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:46.357501 22672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:46.367578 22672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45759
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:46.368947 22672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:46.370335 22672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:46.383431 22678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:46.383453 22679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:46.384107 22681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:46.384495 22672 server_base.cc:1034] running on GCE node
I20260809 05:26:46.385375 22672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:46.386979 22672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:46.388216 22672 hybrid_clock.cc:648] HybridClock initialized: now 1786253206388161 us; error 53 us; skew 500 ppm
I20260809 05:26:46.388612 22672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:46 dist-test-slave-rmhg krb5kdc[22653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:46.391896 22672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260809 05:26:46.393465 22672 webserver.cc:460] Webserver started at http://127.0.0.1:33037/ using document root <none> and password file <none>
I20260809 05:26:46.394187 22672 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:46.394311 22672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:46.394629 22672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:46.397840 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "a31f0f546ea540d3988b917ea3eb0357"
format_stamp: "Formatted at 2026-08-09 05:26:46 on dist-test-slave-rmhg"
server_key: "e7a02fdf6152d855ab2b1611dd16b625"
server_key_iv: "d220b811327fd153c6f71703f1ea49aa"
server_key_version: "encryptionkey@0"
I20260809 05:26:46.398638 22672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "a31f0f546ea540d3988b917ea3eb0357"
format_stamp: "Formatted at 2026-08-09 05:26:46 on dist-test-slave-rmhg"
server_key: "e7a02fdf6152d855ab2b1611dd16b625"
server_key_iv: "d220b811327fd153c6f71703f1ea49aa"
server_key_version: "encryptionkey@0"
I20260809 05:26:46.404263 22672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:46.407992 22694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:46.409507 22672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:46.409685 22672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "a31f0f546ea540d3988b917ea3eb0357"
format_stamp: "Formatted at 2026-08-09 05:26:46 on dist-test-slave-rmhg"
server_key: "e7a02fdf6152d855ab2b1611dd16b625"
server_key_iv: "d220b811327fd153c6f71703f1ea49aa"
server_key_version: "encryptionkey@0"
I20260809 05:26:46.409878 22672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:46.415644 22672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:46.420049 22672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:46.420343 22672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:46.438143 22672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45759
I20260809 05:26:46.438185 22767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45759 every 8 connection(s)
I20260809 05:26:46.439587 22672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:46.443311 22769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:46.447675 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22672
I20260809 05:26:46.447835 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:46.448284 22533 external_mini_cluster.cc:1428] Setting key cd8a05f54b78f27f81013c3bf73c9c0f
I20260809 05:26:46.455792 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357: Bootstrap starting.
I20260809 05:26:46.457751 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:46.450148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:36583 (local address 127.0.0.1:45759)
0809 05:26:46.450975 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:46.450995 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:46.451054 (+ 59us) server_negotiation.cc:407] Connection header received
0809 05:26:46.453723 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:46.453757 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:46.454243 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:46.454697 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:46.455716 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:46.455724 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:46.455742 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:46.455760 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:46.455837 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:46.455867 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:46.456105 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:46.456232 (+ 127us) server_negotiation.cc:299] Negotiation successful
0809 05:26:46.456447 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":309,"threads_started":1}
I20260809 05:26:46.459735 22769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:46.461043 22769 log.cc:824] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:46.461987 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:46.464176 22769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357: No bootstrap required, opened a new log
I20260809 05:26:46.469256 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:46.466515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.204:42481 (local address 127.0.0.1:45759)
0809 05:26:46.466740 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:46.466747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:46.467228 (+ 481us) server_negotiation.cc:407] Connection header received
0809 05:26:46.467413 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:46.467416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:46.467478 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:46.467563 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:46.468571 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:46.468575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:46.468578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:46.468595 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:46.468638 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:46.468652 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:46.468746 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:46.468867 (+ 121us) server_negotiation.cc:299] Negotiation successful
0809 05:26:46.468920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:26:46.473466 22769 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260809 05:26:46.473861 22769 raft_consensus.cc:374] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:46.473963 22769 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31f0f546ea540d3988b917ea3eb0357, State: Initialized, Role: FOLLOWER
I20260809 05:26:46.474561 22769 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260809 05:26:46.474799 22769 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:46.474931 22769 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:46.475106 22769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:46.477764 22769 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260809 05:26:46.478255 22769 leader_election.cc:302] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31f0f546ea540d3988b917ea3eb0357; no voters:
I20260809 05:26:46.478820 22769 leader_election.cc:288] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:46.478948 22783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:46.479585 22783 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [term 1 LEADER]: Becoming Leader. State: Replica: a31f0f546ea540d3988b917ea3eb0357, State: Running, Role: LEADER
I20260809 05:26:46.480101 22783 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260809 05:26:46.480901 22769 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:46.482867 22786 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31f0f546ea540d3988b917ea3eb0357. Latest consensus state: current_term: 1 leader_uuid: "a31f0f546ea540d3988b917ea3eb0357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260809 05:26:46.483415 22786 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:46.483569 22784 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a31f0f546ea540d3988b917ea3eb0357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31f0f546ea540d3988b917ea3eb0357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260809 05:26:46.483845 22784 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:46.485265 22795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:46.489619 22795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:46.498653 22795 catalog_manager.cc:1347] Generated new cluster ID: 683d1449f35d4204914d7a9d1bdcb032
I20260809 05:26:46.498780 22795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:46.517052 22795 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:46.517513 22795 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:46.523864 22795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31f0f546ea540d3988b917ea3eb0357: Generated new TSK 0
I20260809 05:26:46.524960 22795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:46.539938 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22672
W20260809 05:26:46.541604 22761 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.204:36583
W20260809 05:26:46.542361 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:46.542498 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22672
2026-08-09T05:26:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:46 dist-test-slave-rmhg krb5kdc[22812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:46 dist-test-slave-rmhg krb5kdc[22812](info): set up 2 sockets
Aug 09 05:26:46 dist-test-slave-rmhg krb5kdc[22812](info): commencing operation
krb5kdc: starting...
W20260809 05:26:47.781472 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.001s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:47 dist-test-slave-rmhg krb5kdc[22812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:48.095580 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43877
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:43877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:48.447984 22832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:48.448287 22832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:48.448419 22832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:48.455682 22832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:48.455801 22832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:48.455878 22832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:48.455952 22832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:48.456009 22832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:48.466022 22832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:43877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:48.467427 22832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:48.468873 22832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:48.480597 22838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:48.480659 22842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:48.480715 22840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:48.482030 22832 server_base.cc:1034] running on GCE node
I20260809 05:26:48.482802 22832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:48.484127 22832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:48.485322 22832 hybrid_clock.cc:648] HybridClock initialized: now 1786253208485281 us; error 29 us; skew 500 ppm
I20260809 05:26:48.485680 22832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:48 dist-test-slave-rmhg krb5kdc[22812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253208, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:48.488612 22832 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:48.490017 22832 webserver.cc:460] Webserver started at http://127.22.1.126:39547/ using document root <none> and password file <none>
I20260809 05:26:48.490644 22832 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:48.490746 22832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:48.490998 22832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:48.493945 22832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "18b4702f953a44a39f90c61a29007ecc"
format_stamp: "Formatted at 2026-08-09 05:26:48 on dist-test-slave-rmhg"
server_key: "994d1946e32432e0e42fad89bd4211bb"
server_key_iv: "e2bc739cc7d415d1a61fa663812d6cc7"
server_key_version: "encryptionkey@0"
I20260809 05:26:48.494678 22832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "18b4702f953a44a39f90c61a29007ecc"
format_stamp: "Formatted at 2026-08-09 05:26:48 on dist-test-slave-rmhg"
server_key: "994d1946e32432e0e42fad89bd4211bb"
server_key_iv: "e2bc739cc7d415d1a61fa663812d6cc7"
server_key_version: "encryptionkey@0"
I20260809 05:26:48.499694 22832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:48.503019 22851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:48.504357 22832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:48.504560 22832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "18b4702f953a44a39f90c61a29007ecc"
format_stamp: "Formatted at 2026-08-09 05:26:48 on dist-test-slave-rmhg"
server_key: "994d1946e32432e0e42fad89bd4211bb"
server_key_iv: "e2bc739cc7d415d1a61fa663812d6cc7"
server_key_version: "encryptionkey@0"
I20260809 05:26:48.504760 22832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:48.516709 22832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:48.520603 22832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:48.520833 22832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:48.537919 22832 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:43877
I20260809 05:26:48.537966 22945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:43877 every 8 connection(s)
I20260809 05:26:48.539292 22832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:48.543097 22946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:48.549633 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 22832
I20260809 05:26:48.549837 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:48.550309 22533 external_mini_cluster.cc:1428] Setting key b367336cc90e18cace0587a397683b91
I20260809 05:26:48.555083 22946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc: Bootstrap starting.
I20260809 05:26:48.559060 22946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:48.560346 22946 log.cc:824] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:48.563616 22946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc: No bootstrap required, opened a new log
Aug 09 05:26:48 dist-test-slave-rmhg krb5kdc[22812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:48.571663 22946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } }
I20260809 05:26:48.572057 22946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:48.572157 22946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18b4702f953a44a39f90c61a29007ecc, State: Initialized, Role: FOLLOWER
I20260809 05:26:48.572849 22946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } }
I20260809 05:26:48.573129 22946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:48.573338 22946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:48.573553 22946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:48.574169 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:48.552340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38781 (local address 127.22.1.126:43877)
0809 05:26:48.553175 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:48.553191 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:48.553243 (+ 52us) server_negotiation.cc:407] Connection header received
0809 05:26:48.555593 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:48.555624 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:48.556155 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:48.556606 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:48.557978 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:48.559970 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:48.561585 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:48.562592 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:48.562778 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:48.566673 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:48.566680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:48.566696 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:48.566734 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:48.569741 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:48.570561 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:48.570567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:48.570573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:48.570756 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:48.571206 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:48.571210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:48.571212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:48.571754 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:48.572058 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:48.572589 (+ 531us) server_negotiation.cc:299] Negotiation successful
0809 05:26:48.572892 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":335,"threads_started":1}
I20260809 05:26:48.576422 22946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } }
I20260809 05:26:48.577003 22946 leader_election.cc:302] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18b4702f953a44a39f90c61a29007ecc; no voters:
I20260809 05:26:48.577672 22946 leader_election.cc:288] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:48.577796 22954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:48.578576 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:48.579820 22946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:48.584578 22954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [term 1 LEADER]: Becoming Leader. State: Replica: 18b4702f953a44a39f90c61a29007ecc, State: Running, Role: LEADER
I20260809 05:26:48.585286 22954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } }
I20260809 05:26:48.588492 22956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18b4702f953a44a39f90c61a29007ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } } }
I20260809 05:26:48.588923 22956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:48.589344 22958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18b4702f953a44a39f90c61a29007ecc. Latest consensus state: current_term: 1 leader_uuid: "18b4702f953a44a39f90c61a29007ecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18b4702f953a44a39f90c61a29007ecc" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43877 } } }
I20260809 05:26:48.589684 22958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:48.605536 22973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:48.614617 22973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:48.621639 22951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:48.588389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35287 (local address 127.22.1.126:43877)
0809 05:26:48.588622 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:48.588629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:48.591309 (+ 2680us) server_negotiation.cc:407] Connection header received
0809 05:26:48.591503 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:48.591510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:48.591599 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:48.591701 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:48.593920 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:48.605606 (+ 11686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:48.607636 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:48.608437 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:48.608639 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:48.612755 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:48.612764 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:48.612770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:48.612816 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:48.616910 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:48.618040 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:48.618046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:48.618050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:48.618232 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:48.618838 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:48.618844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:48.618847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:48.619289 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:48.619576 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:48.620960 (+ 1384us) server_negotiation.cc:299] Negotiation successful
0809 05:26:48.621217 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260809 05:26:48.626243 22973 catalog_manager.cc:1347] Generated new cluster ID: 270f837de6eb41ae8e557731f90a5b4b
I20260809 05:26:48.626375 22973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:48.647753 22973 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:48.649416 22973 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:48.656391 22973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18b4702f953a44a39f90c61a29007ecc: Generated new TSK 0
I20260809 05:26:48.657449 22973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:48.671625 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 22832
I20260809 05:26:48.672868 22939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:48.763458 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 22832
W20260809 05:26:48.786413 22810 connection.cc:535] client connection to 127.22.1.126:43877 recv error: Network error: failed to read from TLS socket (remote: 127.22.1.126:43877): Connection reset by peer (error 104)
2026-08-09T05:26:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:48 dist-test-slave-rmhg krb5kdc[22988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:48 dist-test-slave-rmhg krb5kdc[22988](info): set up 2 sockets
Aug 09 05:26:48 dist-test-slave-rmhg krb5kdc[22988](info): commencing operation
krb5kdc: starting...
W20260809 05:26:49.995420 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.008s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:50 dist-test-slave-rmhg krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:50.338368 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:33421
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:50.685065 23006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:50.685410 23006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:50.685555 23006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:50.692709 23006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:50.692838 23006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:50.692921 23006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:50.693011 23006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:50.693074 23006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:50.703032 23006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:33421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:50.704489 23006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:50.706008 23006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:50.717852 23012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:50.717922 23016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:50.717895 23014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:50.718729 23006 server_base.cc:1034] running on GCE node
I20260809 05:26:50.719537 23006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:50.720991 23006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:50.722204 23006 hybrid_clock.cc:648] HybridClock initialized: now 1786253210722157 us; error 60 us; skew 500 ppm
I20260809 05:26:50.722546 23006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:50 dist-test-slave-rmhg krb5kdc[22988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253210, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:50.730059 23006 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:50.731606 23006 webserver.cc:460] Webserver started at http://127.22.1.126:35699/ using document root <none> and password file <none>
I20260809 05:26:50.732307 23006 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:50.732427 23006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:50.732752 23006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:50.735863 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "7605b1abe8cc43f39d0173702d417f8d"
format_stamp: "Formatted at 2026-08-09 05:26:50 on dist-test-slave-rmhg"
server_key: "fd65983e15081cb9ea14f46a546d378a"
server_key_iv: "87a0df5ef6327e33b4bf3d9d54380706"
server_key_version: "encryptionkey@0"
I20260809 05:26:50.736629 23006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "7605b1abe8cc43f39d0173702d417f8d"
format_stamp: "Formatted at 2026-08-09 05:26:50 on dist-test-slave-rmhg"
server_key: "fd65983e15081cb9ea14f46a546d378a"
server_key_iv: "87a0df5ef6327e33b4bf3d9d54380706"
server_key_version: "encryptionkey@0"
I20260809 05:26:50.741742 23006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 05:26:50.745317 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:50.746762 23006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:50.746932 23006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "7605b1abe8cc43f39d0173702d417f8d"
format_stamp: "Formatted at 2026-08-09 05:26:50 on dist-test-slave-rmhg"
server_key: "fd65983e15081cb9ea14f46a546d378a"
server_key_iv: "87a0df5ef6327e33b4bf3d9d54380706"
server_key_version: "encryptionkey@0"
I20260809 05:26:50.747124 23006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:50.772578 23006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:50.776994 23006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:50.777313 23006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:50.795615 23006 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:33421
I20260809 05:26:50.795646 23104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:33421 every 8 connection(s)
I20260809 05:26:50.797101 23006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:50.800628 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23006
I20260809 05:26:50.800827 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:50.801281 23105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:50.801426 22533 external_mini_cluster.cc:1428] Setting key d74fb2143f223693c03ede407e471da0
I20260809 05:26:50.815148 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d: Bootstrap starting.
I20260809 05:26:50.815150 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:50.803227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37939 (local address 127.22.1.126:33421)
0809 05:26:50.804053 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:50.804066 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:50.804109 (+ 43us) server_negotiation.cc:407] Connection header received
0809 05:26:50.806000 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:50.806022 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:50.806417 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:50.806741 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:50.807393 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:50.808894 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:50.810659 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:50.811348 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:50.811506 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:50.812816 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:50.812824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:50.812839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:50.813447 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:50.813539 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:50.813568 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:50.813687 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:50.813946 (+ 259us) server_negotiation.cc:299] Negotiation successful
0809 05:26:50.814131 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":340,"threads_started":1}
I20260809 05:26:50.818396 23105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:50.818955 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:50.819591 23105 log.cc:824] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:50.822872 23105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d: No bootstrap required, opened a new log
I20260809 05:26:50.830644 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:50.823875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38139 (local address 127.22.1.126:33421)
0809 05:26:50.824086 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:50.824091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:50.824540 (+ 449us) server_negotiation.cc:407] Connection header received
0809 05:26:50.824683 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:50.824689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:50.824772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:50.824886 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:50.825388 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:50.826562 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:50.827916 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:50.828627 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:50.828769 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:50.829811 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:50.829818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:50.829823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:50.829846 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:50.829917 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:50.829953 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:50.830081 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:50.830224 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 05:26:50.830290 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260809 05:26:50.832289 23105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } }
I20260809 05:26:50.832681 23105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:50.832757 23105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7605b1abe8cc43f39d0173702d417f8d, State: Initialized, Role: FOLLOWER
I20260809 05:26:50.833339 23105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } }
I20260809 05:26:50.833577 23105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:50.833680 23105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:50.833799 23105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:50.835899 23105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } }
I20260809 05:26:50.836372 23105 leader_election.cc:302] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7605b1abe8cc43f39d0173702d417f8d; no voters:
I20260809 05:26:50.836946 23105 leader_election.cc:288] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:50.837096 23120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:50.838243 23120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [term 1 LEADER]: Becoming Leader. State: Replica: 7605b1abe8cc43f39d0173702d417f8d, State: Running, Role: LEADER
I20260809 05:26:50.838886 23120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } }
I20260809 05:26:50.839205 23105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:50.841719 23123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7605b1abe8cc43f39d0173702d417f8d. Latest consensus state: current_term: 1 leader_uuid: "7605b1abe8cc43f39d0173702d417f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } } }
I20260809 05:26:50.841626 23122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7605b1abe8cc43f39d0173702d417f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7605b1abe8cc43f39d0173702d417f8d" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33421 } } }
I20260809 05:26:50.842173 23123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:50.842366 23122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:50.844624 23132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:50.849996 23132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:50.858702 23132 catalog_manager.cc:1347] Generated new cluster ID: 05e6c8bbb37c40a2a41c496f99572afb
I20260809 05:26:50.858822 23132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:50.869462 23132 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:50.870791 23132 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:50.879362 23132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7605b1abe8cc43f39d0173702d417f8d: Generated new TSK 0
I20260809 05:26:50.880223 23132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:50.889312 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23006
W20260809 05:26:50.890574 23100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37939
W20260809 05:26:50.891323 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23006: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:50.891489 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23006
W20260809 05:26:50.913242 22985 connection.cc:535] client connection to 127.22.1.126:33421 recv error: Network error: failed to read from TLS socket (remote: 127.22.1.126:33421): Connection reset by peer (error 104)
2026-08-09T05:26:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:51 dist-test-slave-rmhg krb5kdc[23144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:51 dist-test-slave-rmhg krb5kdc[23144](info): set up 2 sockets
Aug 09 05:26:51 dist-test-slave-rmhg krb5kdc[23144](info): commencing operation
krb5kdc: starting...
W20260809 05:26:52.103716 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.001s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:52 dist-test-slave-rmhg krb5kdc[23144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:52.426484 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38965
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:46611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:38965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:52.768122 23166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:52.768409 23166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:52.768502 23166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:52.775590 23166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:52.775702 23166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:52.775756 23166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:52.775816 23166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:52.775859 23166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:52.785574 23166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:46611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:38965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:38965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:52.786854 23166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:52.788208 23166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:52.799932 23172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:52.800064 23171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:52.799974 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:52.801074 23166 server_base.cc:1034] running on GCE node
I20260809 05:26:52.801893 23166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:52.803283 23166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:52.804519 23166 hybrid_clock.cc:648] HybridClock initialized: now 1786253212804447 us; error 35 us; skew 500 ppm
I20260809 05:26:52.804863 23166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:52 dist-test-slave-rmhg krb5kdc[23144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253212, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:52.807938 23166 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:52.809428 23166 webserver.cc:460] Webserver started at http://127.22.1.126:33641/ using document root <none> and password file <none>
I20260809 05:26:52.810110 23166 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:52.810227 23166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:52.810534 23166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:52.813637 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "fcdf5784176d49d7bbce9879dbce332a"
format_stamp: "Formatted at 2026-08-09 05:26:52 on dist-test-slave-rmhg"
server_key: "f8d1ac31126c903980c904d8e9b5c808"
server_key_iv: "3d55e4792206073a4a437bcfb1a2fa76"
server_key_version: "encryptionkey@0"
I20260809 05:26:52.814379 23166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "fcdf5784176d49d7bbce9879dbce332a"
format_stamp: "Formatted at 2026-08-09 05:26:52 on dist-test-slave-rmhg"
server_key: "f8d1ac31126c903980c904d8e9b5c808"
server_key_iv: "3d55e4792206073a4a437bcfb1a2fa76"
server_key_version: "encryptionkey@0"
I20260809 05:26:52.819499 23166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260809 05:26:52.822889 23187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:52.824199 23166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260809 05:26:52.824353 23166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "fcdf5784176d49d7bbce9879dbce332a"
format_stamp: "Formatted at 2026-08-09 05:26:52 on dist-test-slave-rmhg"
server_key: "f8d1ac31126c903980c904d8e9b5c808"
server_key_iv: "3d55e4792206073a4a437bcfb1a2fa76"
server_key_version: "encryptionkey@0"
I20260809 05:26:52.824560 23166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:52.830273 23166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:52.834339 23166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:52.834651 23166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:52.851517 23166 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:38965
I20260809 05:26:52.851545 23256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:38965 every 8 connection(s)
I20260809 05:26:52.852854 23166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:52.856457 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:52.858554 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23166
I20260809 05:26:52.858732 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:52.859261 22533 external_mini_cluster.cc:1428] Setting key d2fb861b3846ba13aae32ef2c39fe222
I20260809 05:26:52.866874 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:52.861048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55299 (local address 127.22.1.126:38965)
0809 05:26:52.861870 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:52.861882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:52.861923 (+ 41us) server_negotiation.cc:407] Connection header received
0809 05:26:52.863576 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:52.863600 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:52.863927 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:52.864243 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:52.865310 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:52.865316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:52.865329 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:52.865343 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:52.865406 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:52.865426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:52.865606 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:52.865712 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 05:26:52.865912 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":355,"threads_started":1}
I20260809 05:26:52.868264 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a: Bootstrap starting.
I20260809 05:26:52.870712 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:52.871670 23257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:52.872926 23257 log.cc:824] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:52.876248 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a: No bootstrap required, opened a new log
I20260809 05:26:52.879829 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:52.875856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48813 (local address 127.22.1.126:38965)
0809 05:26:52.876048 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:52.876054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:52.876461 (+ 407us) server_negotiation.cc:407] Connection header received
0809 05:26:52.876609 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:52.876614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:52.876689 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:52.876801 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:52.878666 (+ 1865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:52.878673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:52.878679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:52.878701 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:52.878759 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:52.878781 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:52.878895 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:52.879203 (+ 308us) server_negotiation.cc:299] Negotiation successful
0809 05:26:52.879310 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:26:52.886134 23257 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } }
I20260809 05:26:52.886762 23257 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:52.886895 23257 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcdf5784176d49d7bbce9879dbce332a, State: Initialized, Role: FOLLOWER
I20260809 05:26:52.887727 23257 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } }
I20260809 05:26:52.888058 23257 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:52.888263 23257 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:52.888463 23257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:52.891562 23257 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } }
I20260809 05:26:52.892132 23257 leader_election.cc:302] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcdf5784176d49d7bbce9879dbce332a; no voters:
I20260809 05:26:52.892866 23257 leader_election.cc:288] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:52.893011 23274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:52.893999 23274 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [term 1 LEADER]: Becoming Leader. State: Replica: fcdf5784176d49d7bbce9879dbce332a, State: Running, Role: LEADER
I20260809 05:26:52.894593 23274 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } }
I20260809 05:26:52.895299 23257 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:52.897365 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcdf5784176d49d7bbce9879dbce332a. Latest consensus state: current_term: 1 leader_uuid: "fcdf5784176d49d7bbce9879dbce332a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } } }
I20260809 05:26:52.897904 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:52.897945 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcdf5784176d49d7bbce9879dbce332a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcdf5784176d49d7bbce9879dbce332a" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 38965 } } }
I20260809 05:26:52.898605 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:52.900059 23284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:52.916172 23284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:52.931547 23284 catalog_manager.cc:1347] Generated new cluster ID: 45b6857c3d554abfb6ef0c9a4ee3b959
I20260809 05:26:52.931674 23284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:52.951586 23284 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:52.952034 23284 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:52.957774 23284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcdf5784176d49d7bbce9879dbce332a: Generated new TSK 0
I20260809 05:26:52.958552 23284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:52.964207 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23166
W20260809 05:26:52.965364 23250 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55299
W20260809 05:26:52.966003 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:52.966162 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23166
2026-08-09T05:26:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2074 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:53 dist-test-slave-rmhg krb5kdc[23302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:53 dist-test-slave-rmhg krb5kdc[23302](info): set up 2 sockets
Aug 09 05:26:53 dist-test-slave-rmhg krb5kdc[23302](info): commencing operation
krb5kdc: starting...
W20260809 05:26:54.184878 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:54 dist-test-slave-rmhg krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:54.502857 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43351
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:41387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:54.850653 23321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:54.850963 23321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:54.851056 23321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:54.858381 23321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:54.858511 23321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:54.858557 23321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:54.858618 23321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:54.858660 23321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:54.868723 23321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:41387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:54.870126 23321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:54.871465 23321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:54.883055 23329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:54.883204 23328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:54.883152 23331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:54.884207 23321 server_base.cc:1034] running on GCE node
I20260809 05:26:54.884999 23321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:54.886427 23321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:54.887665 23321 hybrid_clock.cc:648] HybridClock initialized: now 1786253214887621 us; error 28 us; skew 500 ppm
I20260809 05:26:54.888025 23321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:54 dist-test-slave-rmhg krb5kdc[23302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253214, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:54.891101 23321 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:54.892580 23321 webserver.cc:460] Webserver started at http://127.22.1.126:40111/ using document root <none> and password file <none>
I20260809 05:26:54.893316 23321 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:54.893453 23321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:54.893757 23321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:54.896952 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "6548d03cad224beeba63688fda9a2762"
format_stamp: "Formatted at 2026-08-09 05:26:54 on dist-test-slave-rmhg"
server_key: "62db40f5e852417e71b2fe950b5ecc70"
server_key_iv: "c247f40499a58669f4c75406e54804b4"
server_key_version: "encryptionkey@0"
I20260809 05:26:54.897861 23321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "6548d03cad224beeba63688fda9a2762"
format_stamp: "Formatted at 2026-08-09 05:26:54 on dist-test-slave-rmhg"
server_key: "62db40f5e852417e71b2fe950b5ecc70"
server_key_iv: "c247f40499a58669f4c75406e54804b4"
server_key_version: "encryptionkey@0"
I20260809 05:26:54.903702 23321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:26:54.907644 23340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:54.909130 23321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:26:54.909365 23321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "6548d03cad224beeba63688fda9a2762"
format_stamp: "Formatted at 2026-08-09 05:26:54 on dist-test-slave-rmhg"
server_key: "62db40f5e852417e71b2fe950b5ecc70"
server_key_iv: "c247f40499a58669f4c75406e54804b4"
server_key_version: "encryptionkey@0"
I20260809 05:26:54.909611 23321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:54.931574 23321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:54.935663 23321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:54.935946 23321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:54.954059 23321 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:43351
I20260809 05:26:54.954102 23414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:43351 every 8 connection(s)
I20260809 05:26:54.955576 23321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:54.959367 23415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:54.965982 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23321
I20260809 05:26:54.966172 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:54.966605 22533 external_mini_cluster.cc:1428] Setting key 48f16adfc2786b545b98d4bf2174e65a
I20260809 05:26:54.969779 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762: Bootstrap starting.
I20260809 05:26:54.973170 23415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:54.974328 23415 log.cc:824] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:54.977504 23415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762: No bootstrap required, opened a new log
Aug 09 05:26:54 dist-test-slave-rmhg krb5kdc[23302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:26:54.987449 23415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } }
I20260809 05:26:54.987912 23415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:54.988025 23415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6548d03cad224beeba63688fda9a2762, State: Initialized, Role: FOLLOWER
I20260809 05:26:54.988644 23415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } }
I20260809 05:26:54.988899 23415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:54.989063 23415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:54.989253 23415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:54.991240 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:54.968453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43585 (local address 127.22.1.126:43351)
0809 05:26:54.969255 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:54.969267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:54.969308 (+ 41us) server_negotiation.cc:407] Connection header received
0809 05:26:54.971026 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:54.971050 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:54.971453 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:54.971766 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:54.973644 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:54.975114 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:54.976820 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:54.977607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:54.977770 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:54.982440 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:54.982453 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:54.982475 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:54.982539 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:54.986561 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:54.987289 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:54.987297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:54.987305 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:54.987537 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:54.987936 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:54.987942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:54.987946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:54.988690 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:54.989003 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:54.989615 (+ 612us) server_negotiation.cc:299] Negotiation successful
0809 05:26:54.990099 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":350,"threads_started":1}
I20260809 05:26:54.991854 23415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } }
I20260809 05:26:54.992354 23415 leader_election.cc:302] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6548d03cad224beeba63688fda9a2762; no voters:
I20260809 05:26:54.992995 23415 leader_election.cc:288] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:54.993114 23422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:54.993827 23422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [term 1 LEADER]: Becoming Leader. State: Replica: 6548d03cad224beeba63688fda9a2762, State: Running, Role: LEADER
I20260809 05:26:54.994441 23422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } }
I20260809 05:26:54.995328 23415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:54.995520 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:54.997279 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6548d03cad224beeba63688fda9a2762. Latest consensus state: current_term: 1 leader_uuid: "6548d03cad224beeba63688fda9a2762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } } }
I20260809 05:26:54.997841 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:54.997771 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6548d03cad224beeba63688fda9a2762" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6548d03cad224beeba63688fda9a2762" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 43351 } } }
I20260809 05:26:55.003077 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:55.008756 23431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:55.015235 23431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:55.033486 23431 catalog_manager.cc:1347] Generated new cluster ID: d6c239cbf80343a6a9c7ca93c1438aef
I20260809 05:26:55.033628 23431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:55.035270 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:55.010322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47221 (local address 127.22.1.126:43351)
0809 05:26:55.010527 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:55.010535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:55.017337 (+ 6802us) server_negotiation.cc:407] Connection header received
0809 05:26:55.017843 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:55.017850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:55.017939 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:55.018016 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:55.020134 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:55.021310 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:55.023199 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:55.023851 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:55.023955 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:55.027583 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:55.027591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:55.027597 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:26:55.027638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:55.031558 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:55.032513 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:55.032519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:55.032522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:55.032683 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:26:55.033212 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:26:55.033218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:26:55.033222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:26:55.033725 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:55.033977 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:55.034661 (+ 684us) server_negotiation.cc:299] Negotiation successful
0809 05:26:55.034900 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:26:55.046324 23431 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:55.047717 23431 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:55.060312 23431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6548d03cad224beeba63688fda9a2762: Generated new TSK 0
I20260809 05:26:55.061259 23431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:55.070627 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23321
I20260809 05:26:55.071861 23406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:26:55.162155 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23321
2026-08-09T05:26:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:55 dist-test-slave-rmhg krb5kdc[23457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:55 dist-test-slave-rmhg krb5kdc[23457](info): set up 2 sockets
Aug 09 05:26:55 dist-test-slave-rmhg krb5kdc[23457](info): commencing operation
krb5kdc: starting...
W20260809 05:26:56.388785 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:56 dist-test-slave-rmhg krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:56.713003 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39849
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:45903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:39849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:57.052634 23474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:57.052942 23474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:57.053073 23474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:57.060703 23474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:57.060858 23474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:57.060942 23474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:57.061012 23474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:57.061081 23474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:57.071295 23474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:45903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:39849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:39849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:57.072742 23474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:57.074275 23474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:57.086364 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:57.086158 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:57.086181 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:57.087328 23474 server_base.cc:1034] running on GCE node
I20260809 05:26:57.088167 23474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:57.089710 23474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:57.090916 23474 hybrid_clock.cc:648] HybridClock initialized: now 1786253217090866 us; error 47 us; skew 500 ppm
I20260809 05:26:57.091272 23474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:57 dist-test-slave-rmhg krb5kdc[23457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253217, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:57.097714 23474 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:57.099169 23474 webserver.cc:460] Webserver started at http://127.22.1.126:42349/ using document root <none> and password file <none>
I20260809 05:26:57.099829 23474 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:57.099932 23474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:57.100200 23474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:57.103243 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "b6fbe344924b4957875e49c008bbabf5"
format_stamp: "Formatted at 2026-08-09 05:26:57 on dist-test-slave-rmhg"
server_key: "ec9a4f7eca244724469b36d13b0f5204"
server_key_iv: "74d83044281465123a5675e69feebac5"
server_key_version: "encryptionkey@0"
I20260809 05:26:57.103973 23474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "b6fbe344924b4957875e49c008bbabf5"
format_stamp: "Formatted at 2026-08-09 05:26:57 on dist-test-slave-rmhg"
server_key: "ec9a4f7eca244724469b36d13b0f5204"
server_key_iv: "74d83044281465123a5675e69feebac5"
server_key_version: "encryptionkey@0"
I20260809 05:26:57.109424 23474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260809 05:26:57.113245 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:57.114842 23474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:26:57.115031 23474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "b6fbe344924b4957875e49c008bbabf5"
format_stamp: "Formatted at 2026-08-09 05:26:57 on dist-test-slave-rmhg"
server_key: "ec9a4f7eca244724469b36d13b0f5204"
server_key_iv: "74d83044281465123a5675e69feebac5"
server_key_version: "encryptionkey@0"
I20260809 05:26:57.115232 23474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:57.136139 23474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:57.140560 23474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:57.140841 23474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:57.164073 23474 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:39849
I20260809 05:26:57.164120 23570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:39849 every 8 connection(s)
I20260809 05:26:57.165567 23474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:57.169713 23572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:57.175714 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23474
I20260809 05:26:57.175889 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:57.176326 22533 external_mini_cluster.cc:1428] Setting key c6b06554e00e6d0e6cb11cfb1125782e
I20260809 05:26:57.180642 23572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: Bootstrap starting.
I20260809 05:26:57.184252 23572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:57.185570 23572 log.cc:824] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:57.188982 23572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: No bootstrap required, opened a new log
I20260809 05:26:57.190418 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:57.178202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35271 (local address 127.22.1.126:39849)
0809 05:26:57.178997 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:57.179011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:57.179052 (+ 41us) server_negotiation.cc:407] Connection header received
0809 05:26:57.180703 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:57.180727 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:57.181200 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:57.181566 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:57.182163 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:57.183529 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:57.185377 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:57.186213 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:57.186415 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:57.187470 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:57.187482 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:57.187503 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:57.188237 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:57.188342 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:57.188376 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:57.188500 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:57.188709 (+ 209us) server_negotiation.cc:299] Negotiation successful
0809 05:26:57.189000 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":366,"threads_started":1}
I20260809 05:26:57.194926 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:57.199322 23572 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } }
I20260809 05:26:57.199795 23572 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:57.199939 23572 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6fbe344924b4957875e49c008bbabf5, State: Initialized, Role: FOLLOWER
I20260809 05:26:57.200590 23572 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } }
I20260809 05:26:57.200855 23572 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:57.201002 23572 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:57.201203 23572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:57.203503 23572 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } }
I20260809 05:26:57.204000 23572 leader_election.cc:302] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6fbe344924b4957875e49c008bbabf5; no voters:
I20260809 05:26:57.204644 23572 leader_election.cc:288] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:57.204761 23585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:57.205443 23585 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [term 1 LEADER]: Becoming Leader. State: Replica: b6fbe344924b4957875e49c008bbabf5, State: Running, Role: LEADER
I20260809 05:26:57.206053 23585 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } }
I20260809 05:26:57.206583 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:57.200094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52905 (local address 127.22.1.126:39849)
0809 05:26:57.200284 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:57.200289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:57.200710 (+ 421us) server_negotiation.cc:407] Connection header received
0809 05:26:57.200890 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:57.200894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:57.200961 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:57.201075 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:57.201653 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:57.202456 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:57.203810 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:26:57.204315 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:26:57.204427 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:26:57.205584 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:57.205591 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:57.205596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:57.205617 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:57.205666 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:57.205687 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:57.205765 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:57.206161 (+ 396us) server_negotiation.cc:299] Negotiation successful
0809 05:26:57.206258 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:26:57.207055 23572 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:57.208904 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6fbe344924b4957875e49c008bbabf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } } }
I20260809 05:26:57.209335 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:57.214011 23586 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6fbe344924b4957875e49c008bbabf5. Latest consensus state: current_term: 1 leader_uuid: "b6fbe344924b4957875e49c008bbabf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6fbe344924b4957875e49c008bbabf5" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 39849 } } }
I20260809 05:26:57.214282 23586 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5 [sys.catalog]: This master's current role is: LEADER
W20260809 05:26:57.220530 23605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:26:57.220650 23605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:26:57.220798 23593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:57.222453 23593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:57.231030 23593 catalog_manager.cc:1347] Generated new cluster ID: a4d51d314976494d9f7d5984c8b02bf0
I20260809 05:26:57.231158 23593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:57.254045 23593 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:57.255394 23593 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:57.261808 23593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6fbe344924b4957875e49c008bbabf5: Generated new TSK 0
I20260809 05:26:57.262682 23593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:57.280690 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23474
W20260809 05:26:57.282294 23566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35271
W20260809 05:26:57.283069 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23474: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:57.283211 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23474
2026-08-09T05:26:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2123 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:57 dist-test-slave-rmhg krb5kdc[23612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:57 dist-test-slave-rmhg krb5kdc[23612](info): set up 2 sockets
Aug 09 05:26:57 dist-test-slave-rmhg krb5kdc[23612](info): commencing operation
krb5kdc: starting...
W20260809 05:26:58.503132 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:26:58 dist-test-slave-rmhg krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:26:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:26:58.822682 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35161
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:26:59.166970 23632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:26:59.167279 23632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:26:59.167397 23632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:26:59.174868 23632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:26:59.174991 23632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:26:59.175061 23632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:26:59.175144 23632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:26:59.175203 23632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:26:59.185405 23632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:26:59.186796 23632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:26:59.188153 23632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:26:59.200317 23637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:59.200354 23647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:26:59.200387 23638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:26:59.201220 23632 server_base.cc:1034] running on GCE node
I20260809 05:26:59.202077 23632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:26:59.203586 23632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:26:59.204821 23632 hybrid_clock.cc:648] HybridClock initialized: now 1786253219204776 us; error 40 us; skew 500 ppm
I20260809 05:26:59.205216 23632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:26:59 dist-test-slave-rmhg krb5kdc[23612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253219, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:26:59.208359 23632 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:26:59.209954 23632 webserver.cc:460] Webserver started at http://127.22.1.126:42671/ using document root <none> and password file <none>
I20260809 05:26:59.210644 23632 fs_manager.cc:359] Metadata directory not provided
I20260809 05:26:59.210762 23632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:26:59.211066 23632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:26:59.214118 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "c86e38ee966c4166b2fe22703ec2d26e"
format_stamp: "Formatted at 2026-08-09 05:26:59 on dist-test-slave-rmhg"
server_key: "38b55fa5d76f54733364972d341ce98c"
server_key_iv: "cd3a7d8001476e157fc8dd13fd558014"
server_key_version: "encryptionkey@0"
I20260809 05:26:59.214906 23632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "c86e38ee966c4166b2fe22703ec2d26e"
format_stamp: "Formatted at 2026-08-09 05:26:59 on dist-test-slave-rmhg"
server_key: "38b55fa5d76f54733364972d341ce98c"
server_key_iv: "cd3a7d8001476e157fc8dd13fd558014"
server_key_version: "encryptionkey@0"
I20260809 05:26:59.220201 23632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260809 05:26:59.223745 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:26:59.225236 23632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260809 05:26:59.225399 23632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "c86e38ee966c4166b2fe22703ec2d26e"
format_stamp: "Formatted at 2026-08-09 05:26:59 on dist-test-slave-rmhg"
server_key: "38b55fa5d76f54733364972d341ce98c"
server_key_iv: "cd3a7d8001476e157fc8dd13fd558014"
server_key_version: "encryptionkey@0"
I20260809 05:26:59.225592 23632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:26:59.231326 23632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:26:59.235675 23632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:26:59.235975 23632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:26:59.253417 23632 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35161
I20260809 05:26:59.253456 23729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35161 every 8 connection(s)
I20260809 05:26:59.254849 23632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:26:59.255220 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23632
I20260809 05:26:59.255419 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:26:59.256017 22533 external_mini_cluster.cc:1428] Setting key 129f758ffd457e59194ebd071e36c3a6
I20260809 05:26:59.259547 23730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:26:59.264822 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:59.257905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49219 (local address 127.22.1.126:35161)
0809 05:26:59.258702 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:59.258717 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:59.258767 (+ 50us) server_negotiation.cc:407] Connection header received
0809 05:26:59.261027 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:59.261054 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:59.261498 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:59.261922 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:59.262882 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:59.262889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:59.262906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:59.262923 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:59.263034 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:59.263062 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:59.263295 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:59.263415 (+ 120us) server_negotiation.cc:299] Negotiation successful
0809 05:26:59.263599 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":319,"threads_started":1}
I20260809 05:26:59.268451 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:26:59.271772 23730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e: Bootstrap starting.
I20260809 05:26:59.275522 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:26:59.272737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59005 (local address 127.22.1.126:35161)
0809 05:26:59.272941 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:26:59.272948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:26:59.273485 (+ 537us) server_negotiation.cc:407] Connection header received
0809 05:26:59.273678 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:26:59.273681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:26:59.273736 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:26:59.273828 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:26:59.274920 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:26:59.274924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:26:59.274927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0809 05:26:59.274942 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:26:59.274990 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0809 05:26:59.275005 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:26:59.275093 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 05:26:59.275217 (+ 124us) server_negotiation.cc:299] Negotiation successful
0809 05:26:59.275262 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260809 05:26:59.275813 23730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e: Neither blocks nor log segments found. Creating new log.
I20260809 05:26:59.277024 23730 log.cc:824] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e: Log is configured to *not* fsync() on all Append() calls
I20260809 05:26:59.279985 23730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e: No bootstrap required, opened a new log
I20260809 05:26:59.287178 23730 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } }
I20260809 05:26:59.287526 23730 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260809 05:26:59.287598 23730 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86e38ee966c4166b2fe22703ec2d26e, State: Initialized, Role: FOLLOWER
I20260809 05:26:59.288085 23730 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } }
I20260809 05:26:59.288280 23730 raft_consensus.cc:388] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260809 05:26:59.288374 23730 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260809 05:26:59.288499 23730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:26:59.290640 23730 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } }
I20260809 05:26:59.291184 23730 leader_election.cc:302] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c86e38ee966c4166b2fe22703ec2d26e; no voters:
I20260809 05:26:59.291745 23730 leader_election.cc:288] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260809 05:26:59.291860 23751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:26:59.292491 23751 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [term 1 LEADER]: Becoming Leader. State: Replica: c86e38ee966c4166b2fe22703ec2d26e, State: Running, Role: LEADER
I20260809 05:26:59.293069 23751 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } }
I20260809 05:26:59.293737 23730 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:26:59.295719 23752 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c86e38ee966c4166b2fe22703ec2d26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } } }
I20260809 05:26:59.296136 23752 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:59.295677 23753 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86e38ee966c4166b2fe22703ec2d26e. Latest consensus state: current_term: 1 leader_uuid: "c86e38ee966c4166b2fe22703ec2d26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e38ee966c4166b2fe22703ec2d26e" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35161 } } }
I20260809 05:26:59.296382 23753 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e [sys.catalog]: This master's current role is: LEADER
I20260809 05:26:59.302862 23758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:26:59.308197 23758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260809 05:26:59.318226 23758 catalog_manager.cc:1347] Generated new cluster ID: 0791a2f7ddf84e42978eab78b89d4e2c
I20260809 05:26:59.318351 23758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:26:59.330377 23758 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:26:59.330879 23758 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:26:59.339962 23758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86e38ee966c4166b2fe22703ec2d26e: Generated new TSK 0
I20260809 05:26:59.341038 23758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:26:59.350204 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23632
W20260809 05:26:59.351341 23723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49219
W20260809 05:26:59.352108 22533 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23632: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260809 05:26:59.352255 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23632
2026-08-09T05:26:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2064 ms)
[----------] 18 tests from AuthTokenIssuingTest (38359 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:26:59 dist-test-slave-rmhg krb5kdc[23781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:26:59 dist-test-slave-rmhg krb5kdc[23781](info): set up 2 sockets
Aug 09 05:26:59 dist-test-slave-rmhg krb5kdc[23781](info): commencing operation
krb5kdc: starting...
W20260809 05:27:00.581977 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:27:00 dist-test-slave-rmhg krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:27:00.914541 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:33211
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:27:01.254043 23797 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:01.254405 23797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:01.254508 23797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:01.254592 23797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:01.261708 23797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:01.261803 23797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:01.261857 23797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:01.261909 23797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:01.261969 23797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:01.271951 23797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:01.273355 23797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:01.274834 23797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:01.286633 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:01.286623 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:01.286576 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:01.287405 23797 server_base.cc:1034] running on GCE node
I20260809 05:27:01.288182 23797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:01.289680 23797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:01.290899 23797 hybrid_clock.cc:648] HybridClock initialized: now 1786253221290838 us; error 40 us; skew 500 ppm
I20260809 05:27:01.291226 23797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:01 dist-test-slave-rmhg krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:01.294335 23797 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:27:01.295850 23797 webserver.cc:460] Webserver started at http://127.22.1.126:39743/ using document root <none> and password file <none>
I20260809 05:27:01.296542 23797 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:01.296645 23797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:01.296974 23797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:01.300006 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "ca9be4d28a5c4bebbf77e0d784938953"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "90cfbc20d8868185054917ffbc90e9a9"
server_key_iv: "80e9a0c4a1962257cbf83d68df624362"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.300889 23797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "ca9be4d28a5c4bebbf77e0d784938953"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "90cfbc20d8868185054917ffbc90e9a9"
server_key_iv: "80e9a0c4a1962257cbf83d68df624362"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.306144 23797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260809 05:27:01.309719 23819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:01.311163 23797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260809 05:27:01.311350 23797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "ca9be4d28a5c4bebbf77e0d784938953"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "90cfbc20d8868185054917ffbc90e9a9"
server_key_iv: "80e9a0c4a1962257cbf83d68df624362"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.311573 23797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:01.340193 23797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:01.344283 23797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:01.344563 23797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:01.362018 23797 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:33211
I20260809 05:27:01.362033 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:33211 every 8 connection(s)
I20260809 05:27:01.363353 23797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:27:01.366775 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23797
I20260809 05:27:01.366992 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:27:01.367136 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:01.367704 22533 external_mini_cluster.cc:1428] Setting key bae5960af2acabaf2f633dd596bac383
I20260809 05:27:01.374023 23894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:01 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:01.396908 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.378776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.22.1.126:33211)
0809 05:27:01.379461 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:01.379468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:01.379515 (+ 47us) server_negotiation.cc:407] Connection header received
0809 05:27:01.380642 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:01.380647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:01.380870 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:01.381055 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:01.382352 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.383527 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.384886 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.385486 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.385657 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.389189 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:01.389201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:01.389216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:01.389247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:01.392189 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.393266 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.393274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.393276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.393450 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.393919 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.393923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.393926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.394576 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:01.394741 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:01.395483 (+ 742us) server_negotiation.cc:299] Negotiation successful
0809 05:27:01.395799 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":486,"thread_start_us":320,"threads_started":1}
I20260809 05:27:01.396945 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.377462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:33211 (local address 127.0.0.1:41952)
0809 05:27:01.378254 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.378301 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:01.380294 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:01.381094 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:01.381101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:01.381822 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:01.382153 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.382171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.383670 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.383674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.384752 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.384765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.385717 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.385724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.385955 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.386562 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:01.386586 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:01.388912 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:01.392384 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.392400 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.392416 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.393065 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.393583 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.393588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.393591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.393806 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.394744 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:01.394754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:01.395044 (+ 290us) client_negotiation.cc:769] Sending connection context
0809 05:27:01.395355 (+ 311us) client_negotiation.cc:240] Negotiation successful
0809 05:27:01.395983 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":130,"thread_start_us":312,"threads_started":1}
WARNING: no policy specified for kudu/127.22.1.125@KRBTEST.COM; defaulting to no policy
I20260809 05:27:01.400482 23894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.125@KRBTEST.COM" created.
I20260809 05:27:01.401778 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.401383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54258)
0809 05:27:01.401556 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.401664 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 05:27:01.402129 23821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
I20260809 05:27:01.403616 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.403305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54272)
0809 05:27:01.403419 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.403484 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260809 05:27:01.403805 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42903: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
I20260809 05:27:01.437736 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42903
--webserver_interface=127.22.1.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:27:01.453181 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attempt: 1
I20260809 05:27:01.454479 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.453944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54276)
0809 05:27:01.454180 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.454318 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260809 05:27:01.455591 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.455150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54286)
0809 05:27:01.455309 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.455414 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260809 05:27:01.455780 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42903: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
I20260809 05:27:01.524081 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attempt: 2
I20260809 05:27:01.525460 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.524867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54288)
0809 05:27:01.525137 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.525285 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 05:27:01.526510 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.526106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54294)
0809 05:27:01.526258 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.526343 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 05:27:01.526712 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42903: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
I20260809 05:27:01.617997 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attempt: 3
I20260809 05:27:01.619297 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.618754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54310)
0809 05:27:01.618996 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.619114 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260809 05:27:01.620435 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.620017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54316)
0809 05:27:01.620171 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.620260 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 05:27:01.620654 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42903: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
I20260809 05:27:01.763975 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attempt: 4
I20260809 05:27:01.765417 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.764726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54332)
0809 05:27:01.765106 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.765248 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260809 05:27:01.766479 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.766079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54344)
0809 05:27:01.766225 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.766327 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 05:27:01.766708 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:42903: Network error: Client connection negotiation failed: client connection to 127.22.1.125:42903: connect: Connection refused (error 111)
W20260809 05:27:01.792881 23906 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:01.793301 23906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:01.793398 23906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:01.793483 23906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:01.800966 23906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:01.801071 23906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:01.801129 23906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:01.801216 23906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:01.801296 23906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:01.811513 23906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:42903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.1.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:01.812815 23906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:01.814299 23906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:01.825805 23912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:01.825737 23919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:01.825774 23913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:01.826902 23906 server_base.cc:1034] running on GCE node
I20260809 05:27:01.827653 23906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:01.829027 23906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:01.830286 23906 hybrid_clock.cc:648] HybridClock initialized: now 1786253221830220 us; error 38 us; skew 500 ppm
I20260809 05:27:01.830591 23906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:01 dist-test-slave-rmhg krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:01.833818 23906 init.cc:377] Logged in from keytab as kudu/127.22.1.125@KRBTEST.COM (short username kudu)
I20260809 05:27:01.835419 23906 webserver.cc:460] Webserver started at http://127.22.1.125:37055/ using document root <none> and password file <none>
I20260809 05:27:01.836138 23906 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:01.836231 23906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:01.836513 23906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:01.839608 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data/instance:
uuid: "87c65a17fb1e46248a64051dadde57fc"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "b7a7c915978fd4bfa9f63f86251e1630"
server_key_iv: "819c541681a0e835aebaf02ac7083349"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.840373 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal/instance:
uuid: "87c65a17fb1e46248a64051dadde57fc"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "b7a7c915978fd4bfa9f63f86251e1630"
server_key_iv: "819c541681a0e835aebaf02ac7083349"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.845659 23906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 05:27:01.849192 23930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:01.850510 23906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:27:01.850669 23906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "87c65a17fb1e46248a64051dadde57fc"
format_stamp: "Formatted at 2026-08-09 05:27:01 on dist-test-slave-rmhg"
server_key: "b7a7c915978fd4bfa9f63f86251e1630"
server_key_iv: "819c541681a0e835aebaf02ac7083349"
server_key_version: "encryptionkey@0"
I20260809 05:27:01.850870 23906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:01.864055 23906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:01.868158 23906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:01.868424 23906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:01.886747 23906 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.125:42903
I20260809 05:27:01.886770 24014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.125:42903 every 8 connection(s)
I20260809 05:27:01.888139 23906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
I20260809 05:27:01.891448 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 23906
I20260809 05:27:01.891667 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-1/wal/instance
I20260809 05:27:01.892185 24015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:01.892197 22533 external_mini_cluster.cc:1428] Setting key 9d8de33fbda5fe9583dc15ac0f343c1a
I20260809 05:27:01.900746 24015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:01 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:01.922405 24022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.903953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.22.1.126:33211)
0809 05:27:01.906387 (+ 2434us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:01.906394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:01.906417 (+ 23us) server_negotiation.cc:407] Connection header received
0809 05:27:01.908086 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:01.908092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:01.908180 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:01.908276 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:01.909886 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.910663 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.912188 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.912630 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.912743 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.916739 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:01.916744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:01.916747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:01.916786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:01.919318 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.920048 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.920053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.920055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.920178 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.920589 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.920592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.920595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.920875 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:01.921023 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:01.921798 (+ 775us) server_negotiation.cc:299] Negotiation successful
0809 05:27:01.922025 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2304,"thread_start_us":2178,"threads_started":1}
WARNING: no policy specified for kudu/127.22.1.124@KRBTEST.COM; defaulting to no policy
I20260809 05:27:01.923454 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.904364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:33211 (local address 127.0.0.1:41966)
0809 05:27:01.905675 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.905727 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:01.907803 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:01.908527 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:01.908536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:01.909057 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:01.909682 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.909699 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.910819 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.910823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.912044 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.912056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.912868 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.912876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.913104 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.913890 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:01.913916 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:01.916497 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:01.919461 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.919471 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.919490 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.919920 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.920283 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.920289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.920293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.920472 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.921010 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:01.921027 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:01.921429 (+ 402us) client_negotiation.cc:769] Sending connection context
0809 05:27:01.921681 (+ 252us) client_negotiation.cc:240] Negotiation successful
0809 05:27:01.922251 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":829,"threads_started":1}
I20260809 05:27:01.925297 24015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.124@KRBTEST.COM" created.
Aug 09 05:27:01 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:01.939641 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.926097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54356)
0809 05:27:01.926266 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.926284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:01.926399 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:01.927437 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:01.927441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:01.927814 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:01.927959 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.927967 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.929749 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.929753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.930592 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:01.930605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.931597 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.931601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:01.931728 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.932266 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:01.932282 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:01.934397 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:01.937377 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.937382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.937385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.937819 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.938160 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:01.938164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:01.938167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:01.938335 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:01.938899 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:01.938904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:01.939055 (+ 151us) client_negotiation.cc:769] Sending connection context
0809 05:27:01.939178 (+ 123us) client_negotiation.cc:240] Negotiation successful
0809 05:27:01.939376 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260809 05:27:01.940022 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.926303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54356 (local address 127.22.1.125:42903)
0809 05:27:01.926933 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:01.926940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:01.926970 (+ 30us) server_negotiation.cc:407] Connection header received
0809 05:27:01.927027 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:01.927046 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:01.927280 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:01.927435 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:01.928126 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.929622 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.930724 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:01.931451 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:01.931570 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:01.934613 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:01.934623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:01.934637 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:01.934665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:01.937260 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.937947 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.937954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.937956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.938068 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:01.938431 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:01.938435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:01.938437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:01.938768 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:01.938924 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:01.939371 (+ 447us) server_negotiation.cc:299] Negotiation successful
0809 05:27:01.939652 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":333,"threads_started":1}
I20260809 05:27:01.942596 24015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:27:01.943742 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.943289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60900)
0809 05:27:01.943477 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.943600 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 05:27:01.944057 23936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:01.945603 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.945297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60904)
0809 05:27:01.945421 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.945500 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 05:27:01.945794 24015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
I20260809 05:27:01.963605 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:34945
--webserver_interface=127.22.1.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:40011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:27:01.995113 24015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 1
I20260809 05:27:01.996336 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.995875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60906)
0809 05:27:01.996077 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.996186 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 05:27:01.997351 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:01.996960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60908)
0809 05:27:01.997137 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:01.997242 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260809 05:27:01.997555 24015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.065788 24015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 2
I20260809 05:27:02.065984 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } attempt: 5
I20260809 05:27:02.067132 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.066481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60920)
0809 05:27:02.066827 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.066932 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260809 05:27:02.068063 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.067727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60932)
0809 05:27:02.067873 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.067941 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 05:27:02.068361 24015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:02.080766 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.066625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54368)
0809 05:27:02.066852 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.066867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.066995 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.067490 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.067495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.068023 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.068273 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.068286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.069768 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.069773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.070644 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.070663 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.071615 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.071619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.071781 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.072365 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.072384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.074812 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.078506 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.078510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.078513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.078960 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.079332 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.079335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.079338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.079459 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.080029 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.080042 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.080172 (+ 130us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.080302 (+ 130us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.080515 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260809 05:27:02.080976 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.066895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.22.1.125:42903)
0809 05:27:02.067074 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.067080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.067105 (+ 25us) server_negotiation.cc:407] Connection header received
0809 05:27:02.067176 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.067181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.067291 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.068225 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.068464 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.069635 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.070805 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.071463 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.071601 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.075064 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.075072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.075076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.075120 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.078367 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.079082 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.079088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.079092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.079238 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.079557 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.079562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.079565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.079935 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.080083 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.080424 (+ 341us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.080685 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:27:02.082077 23894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:27:02.083128 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.082740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60942)
0809 05:27:02.082907 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.082982 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 05:27:02.084076 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.083738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60952)
0809 05:27:02.083872 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.083949 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 05:27:02.084265 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.135485 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 1
I20260809 05:27:02.136747 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.136214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60964)
0809 05:27:02.136422 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.136548 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260809 05:27:02.137725 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.137398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60980)
0809 05:27:02.137541 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.137612 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260809 05:27:02.137925 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.159587 24015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 3
I20260809 05:27:02.164405 24023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.163842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60986)
0809 05:27:02.164134 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.164227 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260809 05:27:02.166121 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.165016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60988)
0809 05:27:02.165726 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.165901 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":330,"threads_started":1}
W20260809 05:27:02.166337 24015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.206125 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 2
I20260809 05:27:02.207288 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.206807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:60996)
0809 05:27:02.207028 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.207120 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 05:27:02.208288 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.207958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32780)
0809 05:27:02.208109 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.208178 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 05:27:02.208504 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.309576 24015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 4
I20260809 05:27:02.310717 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.310197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32782)
0809 05:27:02.310418 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.310539 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260809 05:27:02.311833 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.311399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32792)
0809 05:27:02.311565 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.311671 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260809 05:27:02.312072 24015 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
I20260809 05:27:02.314710 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 3
I20260809 05:27:02.315760 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.315273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32798)
0809 05:27:02.315488 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.315593 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260809 05:27:02.316789 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.316432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32808)
0809 05:27:02.316615 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.316688 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260809 05:27:02.316676 24031 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:02.316998 23894 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:34945: Network error: Client connection negotiation failed: client connection to 127.22.1.124:34945: connect: Connection refused (error 111)
W20260809 05:27:02.317059 24031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:02.317168 24031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:02.317265 24031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:02.324564 24031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:02.324649 24031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:02.324708 24031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:02.324759 24031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:02.324819 24031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:02.336354 24031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:40011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:33211,127.22.1.125:42903,127.22.1.124:34945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.1.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:02.337705 24031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:02.338968 24031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:02.350598 24041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:02.350852 24048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:02.351310 24040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:02.351311 24031 server_base.cc:1034] running on GCE node
I20260809 05:27:02.352293 24031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:02.353925 24031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:02.355185 24031 hybrid_clock.cc:648] HybridClock initialized: now 1786253222355123 us; error 57 us; skew 500 ppm
I20260809 05:27:02.355547 24031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253222, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:02.363628 24031 init.cc:377] Logged in from keytab as kudu/127.22.1.124@KRBTEST.COM (short username kudu)
I20260809 05:27:02.365375 24031 webserver.cc:460] Webserver started at http://127.22.1.124:35301/ using document root <none> and password file <none>
I20260809 05:27:02.366271 24031 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:02.366395 24031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:02.366777 24031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:02.370793 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data/instance:
uuid: "aef6b675cfcd460bbc4e49d4cf678d0b"
format_stamp: "Formatted at 2026-08-09 05:27:02 on dist-test-slave-rmhg"
server_key: "9fb45acd6abda7088052005ad6a4b644"
server_key_iv: "9f98770e72ce77278bfa1c6d65e7de3e"
server_key_version: "encryptionkey@0"
I20260809 05:27:02.371775 24031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal/instance:
uuid: "aef6b675cfcd460bbc4e49d4cf678d0b"
format_stamp: "Formatted at 2026-08-09 05:27:02 on dist-test-slave-rmhg"
server_key: "9fb45acd6abda7088052005ad6a4b644"
server_key_iv: "9f98770e72ce77278bfa1c6d65e7de3e"
server_key_version: "encryptionkey@0"
I20260809 05:27:02.377363 24031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 05:27:02.381718 24055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:02.383029 24031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:27:02.383185 24031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "aef6b675cfcd460bbc4e49d4cf678d0b"
format_stamp: "Formatted at 2026-08-09 05:27:02 on dist-test-slave-rmhg"
server_key: "9fb45acd6abda7088052005ad6a4b644"
server_key_iv: "9f98770e72ce77278bfa1c6d65e7de3e"
server_key_version: "encryptionkey@0"
I20260809 05:27:02.383391 24031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:02.411798 24031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:02.415733 24031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:02.415992 24031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:02.433986 24031 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.124:34945
I20260809 05:27:02.433998 24132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.124:34945 every 8 connection(s)
I20260809 05:27:02.435446 24031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
I20260809 05:27:02.439356 24133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:02.440809 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24031
I20260809 05:27:02.441004 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786252853919692-22533-0/minicluster-data/master-2/wal/instance
I20260809 05:27:02.441599 22533 external_mini_cluster.cc:1428] Setting key b59e70e740978d22aa782a70fc8e9c6e
I20260809 05:27:02.449239 24133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253222, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:02.465528 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.443570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.22.1.124:34945)
0809 05:27:02.444361 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.444376 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.444421 (+ 45us) server_negotiation.cc:407] Connection header received
0809 05:27:02.446055 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.446078 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.446488 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.446841 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.448621 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.449976 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.451733 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.452799 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.452995 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.457645 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.457653 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.457664 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.457705 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.461002 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.461778 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.461785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.461791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.461951 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.462576 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.462580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.462582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.463115 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.463432 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.463807 (+ 375us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.464146 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":362,"threads_started":1}
I20260809 05:27:02.467393 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.452037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.22.1.126:33211)
0809 05:27:02.452590 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.452597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.453222 (+ 625us) server_negotiation.cc:407] Connection header received
0809 05:27:02.453298 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.453303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.453386 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.453470 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.455113 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.455941 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.457510 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.457928 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.458026 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.461684 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.461692 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.461696 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.461739 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.464634 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.465484 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.465488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.465491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.465631 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.466074 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.466078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.466081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.466406 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.466574 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.466971 (+ 397us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.467131 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":219,"threads_started":1}
I20260809 05:27:02.467535 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.452093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:33211 (local address 127.0.0.1:41968)
0809 05:27:02.452866 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.452914 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.453084 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.453706 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.453712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.454595 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.454953 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.454968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.456096 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.456102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.457356 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.457370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.458121 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.458128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.458309 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.458829 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.458862 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.461482 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.464755 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.464765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.464771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.465355 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.465768 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.465772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.465775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.465945 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.466521 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.466527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.466677 (+ 150us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.466823 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.467136 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":408,"threads_started":1}
I20260809 05:27:02.469064 24133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253222, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:02.482337 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.469776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:42903 (local address 127.0.0.1:54382)
0809 05:27:02.469920 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.469937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.470018 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.470285 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.470287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.470620 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.470758 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.470766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.471725 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.471731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.472631 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.472643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.473472 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.473478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.473641 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.474227 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.474245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.476364 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.479935 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.479939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.479942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.480292 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.480699 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.480704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.480708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.480852 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.481412 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.481417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.481564 (+ 147us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.481689 (+ 125us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.481983 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 05:27:02.482363 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.469819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54382 (local address 127.22.1.125:42903)
0809 05:27:02.470003 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.470009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.470040 (+ 31us) server_negotiation.cc:407] Connection header received
0809 05:27:02.470099 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.470104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.470176 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.470283 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.470909 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.471589 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.472777 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.473310 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.473423 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.476605 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.476610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.476613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.476647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.479806 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.480404 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.480410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.480413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.480589 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.480956 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.480962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.480965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.481298 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.481439 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.481795 (+ 356us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.481998 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260809 05:27:02.482847 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.468795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.22.1.126:33211)
0809 05:27:02.469251 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.469256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.469272 (+ 16us) server_negotiation.cc:407] Connection header received
0809 05:27:02.469308 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.469311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.469373 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.469452 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.470871 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.471564 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.472741 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.473207 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.473313 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.477408 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.477416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.477420 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.477462 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.480378 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.480965 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.480971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.480975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.481104 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.481477 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.481482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.481485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.481749 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.481996 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.482281 (+ 285us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.482614 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308}
I20260809 05:27:02.483865 24133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253222, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:02.494202 23894 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 4
I20260809 05:27:02.497972 24138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.484680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32828)
0809 05:27:02.484829 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.484846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.484932 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.485524 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.485529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.485871 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.486012 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.486020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.486941 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.486945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.487861 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.487872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.488722 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.488726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.488886 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.489456 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.489477 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.491882 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.495176 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.495181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.495186 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.495698 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.496087 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.496092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.496095 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.496257 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.496953 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.496958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.497127 (+ 169us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.497332 (+ 205us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.497614 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260809 05:27:02.498832 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.484891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.22.1.124:34945)
0809 05:27:02.485313 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.485317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.485330 (+ 13us) server_negotiation.cc:407] Connection header received
0809 05:27:02.485364 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.485367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.485412 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.485495 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.486135 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.486809 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.487993 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.488573 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.488696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.492068 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.492075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.492080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.492119 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.495029 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.495825 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.495830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.495834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.495969 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.496369 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.496375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.496378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.496763 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.496906 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.498304 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.498523 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
I20260809 05:27:02.499744 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.484576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.22.1.125:42903)
0809 05:27:02.484729 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.484734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.484776 (+ 42us) server_negotiation.cc:407] Connection header received
0809 05:27:02.484949 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.484954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.485023 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.485099 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.486242 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.486981 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.488238 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.488833 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.488925 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.493548 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.493553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.493556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.493585 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.496287 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.497090 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.497095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.497098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.497317 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.497757 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.497763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.497766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.498097 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.498345 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.499124 (+ 779us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.499521 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:27:02.504845 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:27:02.504930 24133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: Bootstrap starting.
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:27:02.508944 24133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:02.510332 24133 log.cc:824] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:02.507705 23861 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ca9be4d28a5c4bebbf77e0d784938953"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41978
I20260809 05:27:02.513829 24133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: No bootstrap required, opened a new log
I20260809 05:27:02.515729 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.494783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32834)
0809 05:27:02.495051 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.495069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.495173 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.500059 (+ 4886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.500065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.500446 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.500661 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.500675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.502364 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.502369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.503395 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.503409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.504739 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.504745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.504948 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.505516 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.505538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.509735 (+ 4197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.512922 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.512928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.512932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.513346 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.513725 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.513729 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.513732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.513871 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.514646 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.514652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.514825 (+ 173us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.514991 (+ 166us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.515330 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260809 05:27:02.515760 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.494849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32834 (local address 127.22.1.124:34945)
0809 05:27:02.499722 (+ 4873us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.499727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.499753 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:27:02.499799 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.499803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.499879 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.500797 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.500984 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.502237 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.503537 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.504113 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.504232 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.509922 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.509927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.509930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.509960 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.512793 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.513459 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.513465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.513469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.513619 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.514067 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.514073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.514076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.514464 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.514668 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.515117 (+ 449us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.515342 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4754,"thread_start_us":4541,"threads_started":1}
I20260809 05:27:02.521689 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: Bootstrap starting.
I20260809 05:27:02.524216 24133 raft_consensus.cc:348] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.524708 24133 raft_consensus.cc:729] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aef6b675cfcd460bbc4e49d4cf678d0b, State: Initialized, Role: FOLLOWER
I20260809 05:27:02.525239 24133 consensus_queue.cc:260] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.525756 23894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:02.526219 24142 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.526631 24142 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.527094 23894 log.cc:824] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:02.527498 24133 sys_catalog.cc:564] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:02.530465 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: No bootstrap required, opened a new log
W20260809 05:27:02.536801 24157 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:27:02.537006 24157 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:27:02.538173 23894 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.538568 23894 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9be4d28a5c4bebbf77e0d784938953, State: Initialized, Role: FOLLOWER
I20260809 05:27:02.539067 23894 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.540287 24159 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.541000 24159 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.541373 23894 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:02.541564 23861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 05:27:02.541755 23861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:02.545112 23861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.534937 24031 master_runner.cc:186] Error getting master registration for 127.22.1.126:33211: OK, instance_id { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" instance_seqno: 1786253221357091 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:27:02.546175 24031 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 05:27:02.547827 24091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9be4d28a5c4bebbf77e0d784938953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b"
I20260809 05:27:02.548347 24091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:02.552107 24091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9be4d28a5c4bebbf77e0d784938953 in term 1.
I20260809 05:27:02.552583 23861 leader_election.cc:288] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [CANDIDATE]: Term 1 election: Requested vote from peers 87c65a17fb1e46248a64051dadde57fc (127.22.1.125:42903), aef6b675cfcd460bbc4e49d4cf678d0b (127.22.1.124:34945)
I20260809 05:27:02.553026 23824 leader_election.cc:302] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aef6b675cfcd460bbc4e49d4cf678d0b, ca9be4d28a5c4bebbf77e0d784938953; no voters:
I20260809 05:27:02.553957 24159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:27:02.562892 24159 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [term 1 LEADER]: Becoming Leader. State: Replica: ca9be4d28a5c4bebbf77e0d784938953, State: Running, Role: LEADER
I20260809 05:27:02.564257 24159 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.579270 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9be4d28a5c4bebbf77e0d784938953. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.579797 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:02.589802 24184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260809 05:27:02.597424 23827 tablet.cc:2369] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:27:02.611282 24015 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } attempt: 5
I20260809 05:27:02.615289 24184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 09 05:27:02 dist-test-slave-rmhg krb5kdc[23781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253221, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:27:02.624332 24139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.585900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.22.1.124:34945)
0809 05:27:02.586115 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.586122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.599056 (+ 12934us) server_negotiation.cc:407] Connection header received
0809 05:27:02.599609 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.599616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.599735 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.599808 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.602590 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.603803 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.607839 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.608529 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.608654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.615146 (+ 6492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.615154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.615158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.615206 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.619100 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.620844 (+ 1744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.620850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.620854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.621029 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.621939 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.621945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.621949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.622342 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.622594 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.623656 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.623935 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:27:02.631986 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.611883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:34945 (local address 127.0.0.1:32850)
0809 05:27:02.612284 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:02.612304 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:02.612410 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:02.613305 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:02.613311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:02.613826 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:02.614107 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.614119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.615744 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.615749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.617265 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:02.617281 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.618558 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.618565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:02.618746 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.619327 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:02.619351 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:02.622201 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:02.628526 (+ 6325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.628535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.628540 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.629129 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.629669 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:02.629675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:02.629679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:02.629872 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:02.630861 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:02.630866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:02.631106 (+ 240us) client_negotiation.cc:769] Sending connection context
0809 05:27:02.631251 (+ 145us) client_negotiation.cc:240] Negotiation successful
0809 05:27:02.631556 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267}
I20260809 05:27:02.633095 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.612640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.22.1.124:34945)
0809 05:27:02.612842 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.612848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.612866 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:27:02.612922 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.612927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.613010 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.613107 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.614283 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.615459 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.617439 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.618147 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.618276 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.622420 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.622427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.622432 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.622470 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.628250 (+ 5780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.629343 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.629349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.629353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.629532 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.630063 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.630068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.630072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.630628 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.630790 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.632402 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.632665 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260809 05:27:02.639541 24015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: Bootstrap starting.
I20260809 05:27:02.644349 24015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:02.640313 24196 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 05:27:02.645856 24015 log.cc:824] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:02.646875 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.577413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.22.1.126:33211)
0809 05:27:02.577602 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.577609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.591353 (+ 13744us) server_negotiation.cc:407] Connection header received
0809 05:27:02.592747 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.592755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.592859 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.592957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.599903 (+ 6946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.601105 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.606382 (+ 5277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.607115 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.607232 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.635827 (+ 28595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.635834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.635839 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.635889 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.639858 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.643344 (+ 3486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.643349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.643353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.643529 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.644718 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.644724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.644728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.645217 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.645522 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.646140 (+ 618us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.646443 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:27:02.651517 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.584412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.22.1.125:42903)
0809 05:27:02.584640 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.584648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.596802 (+ 12154us) server_negotiation.cc:407] Connection header received
0809 05:27:02.597399 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.597407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.597501 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.597577 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.603202 (+ 5625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.604333 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.610851 (+ 6518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.611526 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.611627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.638723 (+ 27096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:02.638730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:02.638735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:02.638779 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:02.642391 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.648966 (+ 6575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.648973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.648978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.649143 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:02.649661 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:02.649666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:02.649669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:02.650046 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:02.650289 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.650906 (+ 617us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.651169 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260809 05:27:02.654191 24091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [term 1 FOLLOWER]: Refusing update from remote peer ca9be4d28a5c4bebbf77e0d784938953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:27:02.655187 24164 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:27:02.645273 23797 master_runner.cc:186] Error getting master registration for 127.22.1.125:42903: OK, instance_id { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" instance_seqno: 1786253221881536 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:27:02.656085 23797 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 05:27:02.659974 24015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: No bootstrap required, opened a new log
W20260809 05:27:02.668149 24065 tablet.cc:2369] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:27:02.675292 24142 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9be4d28a5c4bebbf77e0d784938953. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.675647 24142 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.678004 24015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.678483 24015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c65a17fb1e46248a64051dadde57fc, State: Initialized, Role: FOLLOWER
I20260809 05:27:02.679065 24015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } }
I20260809 05:27:02.679723 24194 mvcc.cc:205] Tried to move back new op lower bound from 7316493199978639360 to 7316493199636774912. Current Snapshot: MvccSnapshot[applied={T|T < 7316493199978639360}]
I20260809 05:27:02.681427 24201 mvcc.cc:205] Tried to move back new op lower bound from 7316493199978639360 to 7316493199636774912. Current Snapshot: MvccSnapshot[applied={T|T < 7316493199978639360}]
I20260809 05:27:02.681874 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.682133 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:02.683149 24015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:02.684597 24184 catalog_manager.cc:1347] Generated new cluster ID: 3b77e33680b944c8b6c1db496898bd5b
I20260809 05:27:02.684693 24184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:27:02.684919 23966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca9be4d28a5c4bebbf77e0d784938953" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "87c65a17fb1e46248a64051dadde57fc"
I20260809 05:27:02.685475 23965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:02.680382 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.686106 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.689312 23965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [term 1 FOLLOWER]: Refusing update from remote peer ca9be4d28a5c4bebbf77e0d784938953: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:27:02.690153 24159 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:27:02.692719 23966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9be4d28a5c4bebbf77e0d784938953 in term 1.
I20260809 05:27:02.707489 24142 sys_catalog.cc:455] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.707728 24142 sys_catalog.cc:458] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.717434 24212 mvcc.cc:205] Tried to move back new op lower bound from 7316493199978639360 to 7316493199636774912. Current Snapshot: MvccSnapshot[applied={T|T < 7316493199978639360}]
I20260809 05:27:02.718315 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca9be4d28a5c4bebbf77e0d784938953. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.718734 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.719169 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.720474 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:02.727233 24184 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:27:02.729460 24184 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:27:02.731254 24164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca9be4d28a5c4bebbf77e0d784938953" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9be4d28a5c4bebbf77e0d784938953" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 33211 } } peers { permanent_uuid: "87c65a17fb1e46248a64051dadde57fc" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 42903 } } peers { permanent_uuid: "aef6b675cfcd460bbc4e49d4cf678d0b" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 34945 } } }
I20260809 05:27:02.731549 24164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953 [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:02.746557 24184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca9be4d28a5c4bebbf77e0d784938953: Generated new TSK 0
I20260809 05:27:02.748044 24184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:27:02.789005 24230 catalog_manager.cc:1259] Loaded cluster ID: 3b77e33680b944c8b6c1db496898bd5b
I20260809 05:27:02.789130 24230 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: loading cluster ID for follower catalog manager: success
I20260809 05:27:02.793282 24230 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: acquiring CA information for follower catalog manager: success
I20260809 05:27:02.796335 24230 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87c65a17fb1e46248a64051dadde57fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 05:27:02.888892 24137 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:02.881591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.22.1.126:33211)
0809 05:27:02.882361 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.882367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.886784 (+ 4417us) server_negotiation.cc:407] Connection header received
0809 05:27:02.887134 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.887141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.887256 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.887338 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.888603 (+ 1265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41990: BlockingRecv error: recv got EOF from 127.0.0.1:41990 (error 108)
Metrics: {"server-negotiator.queue_time_us":590}
W20260809 05:27:02.895689 24026 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:02.882621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.22.1.125:42903)
0809 05:27:02.882823 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.882829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.893683 (+ 10854us) server_negotiation.cc:407] Connection header received
0809 05:27:02.893943 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.893949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.894032 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.894126 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.895446 (+ 1320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54408: BlockingRecv error: recv got EOF from 127.0.0.1:54408 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260809 05:27:02.906186 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:02.903690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.22.1.124:34945)
0809 05:27:02.903893 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.903900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.904466 (+ 566us) server_negotiation.cc:407] Connection header received
0809 05:27:02.904539 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.904544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.904627 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.904721 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:02.905907 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32858: BlockingRecv error: recv got EOF from 127.0.0.1:32858 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:27:02.948117 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:02.928664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.22.1.125:42903)
0809 05:27:02.928860 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.928867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:02.939428 (+ 10561us) server_negotiation.cc:407] Connection header received
0809 05:27:02.940040 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:02.940047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:02.940275 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:02.940418 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:02.941313 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.942559 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.944923 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:02.945616 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:02.945725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:02.946510 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:02.947281 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:02.947399 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:02.947659 (+ 260us) server_negotiation.cc:299] Negotiation successful
0809 05:27:02.947773 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260809 05:27:03.002430 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:02.994411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.22.1.124:34945)
0809 05:27:02.994627 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:02.994636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.000236 (+ 5600us) server_negotiation.cc:407] Connection header received
0809 05:27:03.000803 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.000810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.000934 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.001003 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.002172 (+ 1169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32868: BlockingRecv error: recv got EOF from 127.0.0.1:32868 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:27:03.049324 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.038342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.22.1.125:42903)
0809 05:27:03.038569 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.038576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.042234 (+ 3658us) server_negotiation.cc:407] Connection header received
0809 05:27:03.042683 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.042690 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.042794 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.042865 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.043642 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.044810 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.046923 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.047648 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.047752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.048406 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.048580 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.048643 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.048891 (+ 248us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.048983 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 05:27:03.108639 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.098138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.22.1.124:34945)
0809 05:27:03.098366 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.098373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.106373 (+ 8000us) server_negotiation.cc:407] Connection header received
0809 05:27:03.107001 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.107008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.107112 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.107197 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.108332 (+ 1135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32884: BlockingRecv error: recv got EOF from 127.0.0.1:32884 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260809 05:27:03.147564 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.135848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.22.1.125:42903)
0809 05:27:03.136041 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.136047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.140972 (+ 4925us) server_negotiation.cc:407] Connection header received
0809 05:27:03.141300 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.141306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.141405 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.141525 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.142063 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.143202 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.145294 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.145952 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.146083 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.146657 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.146806 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.146879 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.147126 (+ 247us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.147216 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260809 05:27:03.159806 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.157145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32896 (local address 127.22.1.124:34945)
0809 05:27:03.157365 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.157372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.158214 (+ 842us) server_negotiation.cc:407] Connection header received
0809 05:27:03.158415 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.158422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.158529 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.158651 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.159536 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32896: BlockingRecv error: recv got EOF from 127.0.0.1:32896 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:27:03.175367 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.168032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.22.1.125:42903)
0809 05:27:03.168247 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.168252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.168833 (+ 581us) server_negotiation.cc:407] Connection header received
0809 05:27:03.169016 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.169022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.169122 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.169263 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.169800 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.171022 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.173116 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.173940 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.174096 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.174579 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.174728 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.174825 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.174964 (+ 139us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.175055 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260809 05:27:03.195639 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.186681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.22.1.124:34945)
0809 05:27:03.186915 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.186921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.193876 (+ 6955us) server_negotiation.cc:407] Connection header received
0809 05:27:03.194123 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.194129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.194254 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.194362 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.195387 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32902: BlockingRecv error: recv got EOF from 127.0.0.1:32902 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
I20260809 05:27:03.214766 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.207513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54458 (local address 127.22.1.125:42903)
0809 05:27:03.207707 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.207714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.208248 (+ 534us) server_negotiation.cc:407] Connection header received
0809 05:27:03.208576 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.208583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.208681 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.208787 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.209336 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.210547 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.212517 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.213278 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.213413 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.213917 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.214083 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.214172 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.214347 (+ 175us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.214432 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260809 05:27:03.224377 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.221940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.22.1.124:34945)
0809 05:27:03.222121 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.222127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.222651 (+ 524us) server_negotiation.cc:407] Connection header received
0809 05:27:03.222913 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.222919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.223016 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.223092 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.224123 (+ 1031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32910: BlockingRecv error: recv got EOF from 127.0.0.1:32910 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:27:03.256752 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.250776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.22.1.125:42903)
0809 05:27:03.250939 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.250943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.251293 (+ 350us) server_negotiation.cc:407] Connection header received
0809 05:27:03.251564 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.251569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.251671 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.251754 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.252243 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.253059 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.255005 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.255502 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.255634 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.256089 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.256190 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.256265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.256431 (+ 166us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.256502 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260809 05:27:03.266948 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.262331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.22.1.124:34945)
0809 05:27:03.262478 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.262483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.265638 (+ 3155us) server_negotiation.cc:407] Connection header received
0809 05:27:03.265807 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.265812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.265885 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.265968 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.266729 (+ 761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32918: BlockingRecv error: recv got EOF from 127.0.0.1:32918 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260809 05:27:03.308509 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.302723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.22.1.125:42903)
0809 05:27:03.302928 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.302933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.303284 (+ 351us) server_negotiation.cc:407] Connection header received
0809 05:27:03.303504 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.303508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.303607 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.303685 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.304142 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.304874 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.306813 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.307255 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.307345 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.307815 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.307919 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.307983 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.308124 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.308188 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260809 05:27:03.317358 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.315023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.22.1.124:34945)
0809 05:27:03.315263 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.315271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.315653 (+ 382us) server_negotiation.cc:407] Connection header received
0809 05:27:03.315947 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.315952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.316029 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.316109 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.317082 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32930: BlockingRecv error: recv got EOF from 127.0.0.1:32930 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:27:03.389437 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.384535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.22.1.125:42903)
0809 05:27:03.384706 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.384711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.385118 (+ 407us) server_negotiation.cc:407] Connection header received
0809 05:27:03.385368 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.385372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.385443 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.385520 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.385850 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.386607 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.387935 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.388382 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.388469 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.388800 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.388913 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.388976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.389073 (+ 97us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.389134 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 05:27:03.396611 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.394846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.22.1.124:34945)
0809 05:27:03.395069 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.395076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.395317 (+ 241us) server_negotiation.cc:407] Connection header received
0809 05:27:03.395483 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.395487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.395580 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.395663 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.396382 (+ 719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32932: BlockingRecv error: recv got EOF from 127.0.0.1:32932 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:27:03.533327 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.528148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.22.1.125:42903)
0809 05:27:03.528367 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.528372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.528639 (+ 267us) server_negotiation.cc:407] Connection header received
0809 05:27:03.528843 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.528847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.528916 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.528990 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.529490 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.530182 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.531572 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.532119 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.532230 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.532621 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.532737 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.532808 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.532957 (+ 149us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.533037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260809 05:27:03.539063 24157 catalog_manager.cc:1259] Loaded cluster ID: 3b77e33680b944c8b6c1db496898bd5b
I20260809 05:27:03.539184 24157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: loading cluster ID for follower catalog manager: success
I20260809 05:27:03.542639 24157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: acquiring CA information for follower catalog manager: success
W20260809 05:27:03.543919 24136 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:03.541274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.22.1.124:34945)
0809 05:27:03.541459 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.541465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.541869 (+ 404us) server_negotiation.cc:407] Connection header received
0809 05:27:03.542031 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.542037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.542692 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.542807 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:03.543673 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32936: BlockingRecv error: recv got EOF from 127.0.0.1:32936 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:27:03.545256 24157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aef6b675cfcd460bbc4e49d4cf678d0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:27:03.808169 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.803471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.22.1.125:42903)
0809 05:27:03.803665 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.803670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.803922 (+ 252us) server_negotiation.cc:407] Connection header received
0809 05:27:03.804083 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.804087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.804159 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.804249 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.804613 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.805360 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.806602 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.807047 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.807140 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.807583 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.807710 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.807790 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.807886 (+ 96us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.807958 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260809 05:27:03.819787 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.814320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.22.1.124:34945)
0809 05:27:03.814537 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.814544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.814656 (+ 112us) server_negotiation.cc:407] Connection header received
0809 05:27:03.814867 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.814872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.815033 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.815153 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.815571 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.816285 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.817617 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.818032 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.818192 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.818666 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.819209 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.819362 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.819437 (+ 75us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.819533 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:27:03.835063 24026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.826080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.22.1.125:42903)
0809 05:27:03.826284 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.826291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.828963 (+ 2672us) server_negotiation.cc:407] Connection header received
0809 05:27:03.829262 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.829268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.829358 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.829444 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.830001 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.831040 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.832808 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.833527 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.833676 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.834175 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.834316 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.834401 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.834639 (+ 238us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.834742 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260809 05:27:03.835981 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.826445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.22.1.124:34945)
0809 05:27:03.826633 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.826638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.829723 (+ 3085us) server_negotiation.cc:407] Connection header received
0809 05:27:03.829886 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.829892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.829970 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.830063 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.830879 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.831894 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.833978 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.834572 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.834683 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.835206 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.835362 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.835452 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.835652 (+ 200us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.835743 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:27:03.836755 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:03.826288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.22.1.126:33211)
0809 05:27:03.828552 (+ 2264us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:03.828558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:03.829280 (+ 722us) server_negotiation.cc:407] Connection header received
0809 05:27:03.829649 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:03.829656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:03.829872 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:03.830055 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:03.830666 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.831732 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.833617 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:03.834229 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:03.834339 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:03.835471 (+ 1132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:03.835986 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:03.836099 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:03.836360 (+ 261us) server_negotiation.cc:299] Negotiation successful
0809 05:27:03.836456 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2128,"thread_start_us":1984,"threads_started":1}
I20260809 05:27:03.841320 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23797
I20260809 05:27:03.842443 23887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:03.949438 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23797
W20260809 05:27:03.974380 23934 connection.cc:535] server connection from 127.0.0.1:54368 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260809 05:27:03.974797 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 23906
I20260809 05:27:03.975883 24007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:04.081719 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 23906
I20260809 05:27:04.108038 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24031
I20260809 05:27:04.109181 24124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:04.214524 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24031
2026-08-09T05:27:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4867 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 09 05:27:04 dist-test-slave-rmhg krb5kdc[24423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 09 05:27:04 dist-test-slave-rmhg krb5kdc[24423](info): set up 2 sockets
Aug 09 05:27:04 dist-test-slave-rmhg krb5kdc[24423](info): commencing operation
krb5kdc: starting...
W20260809 05:27:05.441535 22533 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/test-user.keytab.
Aug 09 05:27:05 dist-test-slave-rmhg krb5kdc[24423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-09T05:27:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-09T05:27:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.1.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
Entry for principal kudu/127.22.1.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab.
I20260809 05:27:05.764719 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35743
--webserver_interface=127.22.1.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260809 05:27:06.106171 24445 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:06.106498 24445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:06.106600 24445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:06.106716 24445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:06.113868 24445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:06.113973 24445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:06.114059 24445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:06.114130 24445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:06.114212 24445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:06.124334 24445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.126.keytab
--principal=kudu/127.22.1.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.126:35743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.1.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:06.125721 24445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:06.127079 24445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:06.138366 24453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:06.138459 24450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:06.138432 24451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:06.139302 24445 server_base.cc:1034] running on GCE node
I20260809 05:27:06.140067 24445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:06.141489 24445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:06.142714 24445 hybrid_clock.cc:648] HybridClock initialized: now 1786253226142638 us; error 47 us; skew 500 ppm
I20260809 05:27:06.143039 24445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:06.146407 24445 init.cc:377] Logged in from keytab as kudu/127.22.1.126@KRBTEST.COM (short username kudu)
I20260809 05:27:06.147845 24445 webserver.cc:460] Webserver started at http://127.22.1.126:39887/ using document root <none> and password file <none>
I20260809 05:27:06.148531 24445 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:06.148623 24445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:06.148909 24445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:06.152006 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data/instance:
uuid: "b4c618aa08604239a571be3b35a63cfa"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "32b2ce26ce19563203bba6f6994877ce"
server_key_iv: "02a31bf0386db05d5f431040ba4f4992"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.152792 24445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance:
uuid: "b4c618aa08604239a571be3b35a63cfa"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "32b2ce26ce19563203bba6f6994877ce"
server_key_iv: "02a31bf0386db05d5f431040ba4f4992"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.158491 24445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260809 05:27:06.162277 24461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:06.163728 24445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260809 05:27:06.163925 24445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal
uuid: "b4c618aa08604239a571be3b35a63cfa"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "32b2ce26ce19563203bba6f6994877ce"
server_key_iv: "02a31bf0386db05d5f431040ba4f4992"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.164151 24445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:06.175801 24445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:06.179984 24445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:06.180258 24445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:06.198190 24445 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.126:35743
I20260809 05:27:06.198213 24541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.126:35743 every 8 connection(s)
I20260809 05:27:06.199689 24445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/data/info.pb
I20260809 05:27:06.203509 24542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:06.206331 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24445
I20260809 05:27:06.206518 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-0/wal/instance
I20260809 05:27:06.206954 22533 external_mini_cluster.cc:1428] Setting key 1898e40ce4337c1829918cdcb3625de4
I20260809 05:27:06.210287 24542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:06.233398 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.214528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.22.1.126:35743)
0809 05:27:06.215110 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:06.215116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:06.215160 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:27:06.216992 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:06.216998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:06.217283 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:06.217446 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:06.218927 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.220114 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.221808 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.222613 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.222802 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.226430 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:06.226440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:06.226459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:06.226497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:06.229142 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.229992 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.229998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.230000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.230170 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.230606 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.230610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.230612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.231150 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:06.231311 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:06.231961 (+ 650us) server_negotiation.cc:299] Negotiation successful
0809 05:27:06.232239 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":422,"thread_start_us":283,"threads_started":1}
I20260809 05:27:06.233446 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.213396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35743 (local address 127.0.0.1:53310)
0809 05:27:06.214289 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.214351 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:06.216607 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:06.217513 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:06.217519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:06.218261 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:06.218722 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.218746 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.220262 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.220267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.221669 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.221682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.222867 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.222873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.223094 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.223716 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:06.223740 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:06.226177 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:06.229320 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.229332 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.229345 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.229864 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.230287 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.230292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.230296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.230508 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.231300 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:06.231309 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:06.231561 (+ 252us) client_negotiation.cc:769] Sending connection context
0809 05:27:06.231835 (+ 274us) client_negotiation.cc:240] Negotiation successful
0809 05:27:06.232400 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":127,"thread_start_us":348,"threads_started":1}
I20260809 05:27:06.236824 24542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:27:06.238142 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.237644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35850)
0809 05:27:06.237840 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.237980 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260809 05:27:06.238458 24468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.22.1.125@KRBTEST.COM; defaulting to no policy
I20260809 05:27:06.240075 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.239653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35858)
0809 05:27:06.239816 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.239907 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260809 05:27:06.240301 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:36083: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
Entry for principal kudu/127.22.1.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab.
I20260809 05:27:06.277612 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:36083
--webserver_interface=127.22.1.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:27:06.289669 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } attempt: 1
I20260809 05:27:06.290915 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.290457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35868)
0809 05:27:06.290674 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.290764 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260809 05:27:06.291960 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.291618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35870)
0809 05:27:06.291756 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.291827 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260809 05:27:06.292157 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:36083: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
I20260809 05:27:06.360379 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } attempt: 2
I20260809 05:27:06.361698 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.361216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35872)
0809 05:27:06.361447 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.361541 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260809 05:27:06.362753 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.362404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35876)
0809 05:27:06.362534 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.362621 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 05:27:06.362944 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:36083: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
I20260809 05:27:06.454193 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } attempt: 3
I20260809 05:27:06.455346 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.454892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35886)
0809 05:27:06.455092 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.455204 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 05:27:06.456472 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.456123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35892)
0809 05:27:06.456263 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.456332 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260809 05:27:06.456674 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:36083: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
I20260809 05:27:06.599916 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } attempt: 4
I20260809 05:27:06.601133 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.600569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35900)
0809 05:27:06.600895 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.600986 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260809 05:27:06.602207 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.601846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35912)
0809 05:27:06.601992 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.602076 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260809 05:27:06.602450 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.125:36083: Network error: Client connection negotiation failed: client connection to 127.22.1.125:36083: connect: Connection refused (error 111)
W20260809 05:27:06.634719 24550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:06.635076 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:06.635150 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:06.635234 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:06.642831 24550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:06.642943 24550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:06.643000 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:06.643050 24550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:06.643115 24550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:06.653399 24550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.125.keytab
--principal=kudu/127.22.1.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.125:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.1.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:06.654794 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:06.656163 24550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:06.668857 24560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:06.668759 24561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:06.668792 24564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:06.669678 24550 server_base.cc:1034] running on GCE node
I20260809 05:27:06.670487 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:06.671931 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:06.673221 24550 hybrid_clock.cc:648] HybridClock initialized: now 1786253226673119 us; error 37 us; skew 500 ppm
I20260809 05:27:06.673576 24550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:06.676985 24550 init.cc:377] Logged in from keytab as kudu/127.22.1.125@KRBTEST.COM (short username kudu)
I20260809 05:27:06.678665 24550 webserver.cc:460] Webserver started at http://127.22.1.125:34717/ using document root <none> and password file <none>
I20260809 05:27:06.679471 24550 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:06.679591 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:06.679927 24550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:06.683233 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data/instance:
uuid: "840635203bbb4cb88d979910e5bf4420"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "68ab2ebbd42ec8656e862930fb60cf5b"
server_key_iv: "436519cd4873abecf6e6755a2ff83a04"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.684047 24550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal/instance:
uuid: "840635203bbb4cb88d979910e5bf4420"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "68ab2ebbd42ec8656e862930fb60cf5b"
server_key_iv: "436519cd4873abecf6e6755a2ff83a04"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.689563 24550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260809 05:27:06.693305 24572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:06.694679 24550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260809 05:27:06.694851 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal
uuid: "840635203bbb4cb88d979910e5bf4420"
format_stamp: "Formatted at 2026-08-09 05:27:06 on dist-test-slave-rmhg"
server_key: "68ab2ebbd42ec8656e862930fb60cf5b"
server_key_iv: "436519cd4873abecf6e6755a2ff83a04"
server_key_version: "encryptionkey@0"
I20260809 05:27:06.695091 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:06.718442 24550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:06.723315 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:06.723603 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:06.741724 24550 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.125:36083
I20260809 05:27:06.741779 24645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.125:36083 every 8 connection(s)
I20260809 05:27:06.743173 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/data/info.pb
I20260809 05:27:06.747260 24647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:06.751178 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24550
I20260809 05:27:06.751348 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-1/wal/instance
I20260809 05:27:06.751816 22533 external_mini_cluster.cc:1428] Setting key 42810491fe04e24f44ac031ad14ae571
I20260809 05:27:06.754269 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.1.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.1.124@KRBTEST.COM" created.
I20260809 05:27:06.786216 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.757244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.22.1.126:35743)
0809 05:27:06.765648 (+ 8404us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:06.765655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:06.768543 (+ 2888us) server_negotiation.cc:407] Connection header received
0809 05:27:06.770769 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:06.770777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:06.770887 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:06.771039 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:06.772746 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.773910 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.775550 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.776166 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.776296 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.780223 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:06.780229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:06.780232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:06.780273 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:06.782944 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.783885 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.783889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.783892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.784015 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.784397 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.784401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.784404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.784767 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:06.784889 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:06.785676 (+ 787us) server_negotiation.cc:299] Negotiation successful
0809 05:27:06.785859 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8250,"thread_start_us":8092,"threads_started":1}
I20260809 05:27:06.787352 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.757904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35743 (local address 127.0.0.1:53316)
0809 05:27:06.768302 (+ 10398us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.768373 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:06.770429 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:06.771260 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:06.771269 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:06.771870 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:06.772502 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.772521 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.774087 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.774091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.775390 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.775405 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.776405 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.776410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.776640 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.777420 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:06.777445 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:06.779959 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:06.783112 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.783122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.783143 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.783778 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.784108 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.784111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.784114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.784306 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.784882 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:06.784888 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:06.785331 (+ 443us) client_negotiation.cc:769] Sending connection context
0809 05:27:06.785580 (+ 249us) client_negotiation.cc:240] Negotiation successful
0809 05:27:06.786029 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9986,"thread_start_us":9716,"threads_started":1}
I20260809 05:27:06.789172 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:06.804836 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.789976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35920)
0809 05:27:06.790191 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.790210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:06.790308 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:06.791375 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:06.791379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:06.791737 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:06.791882 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.791890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.793619 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.793623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.794486 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.794499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.795531 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.795536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.795679 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.796263 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:06.796281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:06.798611 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:06.802229 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.802234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.802238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.802739 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.803096 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.803101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.803104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.803255 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.803873 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:06.803878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:06.804034 (+ 156us) client_negotiation.cc:769] Sending connection context
0809 05:27:06.804169 (+ 135us) client_negotiation.cc:240] Negotiation successful
0809 05:27:06.804472 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260809 05:27:06.804909 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.790209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.22.1.125:36083)
0809 05:27:06.790890 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:06.790897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:06.790927 (+ 30us) server_negotiation.cc:407] Connection header received
0809 05:27:06.790977 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:06.790983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:06.791211 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:06.791345 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:06.792043 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.793490 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.794627 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.795381 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.795506 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.798844 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:06.798863 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:06.798885 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:06.798927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:06.802085 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.802860 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.802873 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.802875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.802992 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.803365 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.803368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.803370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.803740 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:06.803906 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:06.804308 (+ 402us) server_negotiation.cc:299] Negotiation successful
0809 05:27:06.804580 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":517,"thread_start_us":335,"threads_started":1}
I20260809 05:27:06.807564 24647 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:27:06.808651 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.808208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52378)
0809 05:27:06.808366 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.808484 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260809 05:27:06.808887 24576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:06.810433 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.810116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52388)
0809 05:27:06.810252 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.810331 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 05:27:06.810636 24647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
Entry for principal kudu/127.22.1.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab.
I20260809 05:27:06.827796 22533 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
/tmp/dist-test-taskGKhvF4/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:33551
--webserver_interface=127.22.1.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.22.1.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260809 05:27:06.859938 24647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 1
I20260809 05:27:06.861092 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.860630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52396)
0809 05:27:06.860822 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.860931 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 05:27:06.862355 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.861885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52408)
0809 05:27:06.862063 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.862171 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260809 05:27:06.862627 24647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:06.901741 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } attempt: 5
Aug 09 05:27:06 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:06.916011 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.902659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.22.1.125:36083)
0809 05:27:06.902848 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:06.902854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:06.902880 (+ 26us) server_negotiation.cc:407] Connection header received
0809 05:27:06.902967 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:06.902982 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:06.903071 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:06.903201 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:06.904002 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.905114 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.906334 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.907050 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.907186 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.910453 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:06.910458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:06.910462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:06.910498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:06.913190 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.914042 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.914046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.914048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.914178 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:06.914569 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:06.914572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:06.914574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:06.914861 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:06.914977 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:06.915547 (+ 570us) server_negotiation.cc:299] Negotiation successful
0809 05:27:06.915747 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:27:06.916038 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.902496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35934)
0809 05:27:06.902715 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.902736 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:06.902852 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:06.903208 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:06.903212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:06.903645 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:06.903831 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.903840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.905288 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.905293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.906174 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:06.906186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:06.907186 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:06.907190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:06.907353 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:06.907970 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:06.907987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:06.910262 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:06.913350 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.913356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.913359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.913914 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.914280 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:06.914285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:06.914288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:06.914471 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:06.914979 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:06.914985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:06.915169 (+ 184us) client_negotiation.cc:769] Sending connection context
0809 05:27:06.915324 (+ 155us) client_negotiation.cc:240] Negotiation successful
0809 05:27:06.915671 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260809 05:27:06.917431 24542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } has no permanent_uuid. Determining permanent_uuid...
I20260809 05:27:06.918573 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.918143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52424)
0809 05:27:06.918323 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.918404 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 05:27:06.919541 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.919203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52436)
0809 05:27:06.919346 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.919433 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260809 05:27:06.919739 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:06.930860 24647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 2
I20260809 05:27:06.931876 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.931460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52442)
0809 05:27:06.931644 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.931729 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260809 05:27:06.932771 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.932454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52456)
0809 05:27:06.932590 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.932659 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260809 05:27:06.932961 24647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:06.970943 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 1
I20260809 05:27:06.972082 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.971619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52464)
0809 05:27:06.971807 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.971899 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260809 05:27:06.973073 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:06.972744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52466)
0809 05:27:06.972886 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:06.972955 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 05:27:06.973281 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:07.024174 24647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 3
I20260809 05:27:07.025424 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.024909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52482)
0809 05:27:07.025128 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.025261 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260809 05:27:07.026515 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.026074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52496)
0809 05:27:07.026309 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.026390 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260809 05:27:07.026722 24647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:07.041471 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 2
I20260809 05:27:07.042613 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.042124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52508)
0809 05:27:07.042337 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.042442 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260809 05:27:07.043674 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.043330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52514)
0809 05:27:07.043493 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.043566 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260809 05:27:07.043866 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:07.150090 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 3
I20260809 05:27:07.151201 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.150732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52524)
0809 05:27:07.150942 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.151035 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260809 05:27:07.152351 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.151944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52538)
0809 05:27:07.152169 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.152240 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260809 05:27:07.152539 24542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
I20260809 05:27:07.169981 24647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 4
I20260809 05:27:07.171344 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.170760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52552)
0809 05:27:07.171036 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.171149 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260809 05:27:07.174594 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.174226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52568)
0809 05:27:07.174379 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.174470 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260809 05:27:07.174871 24647 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.22.1.124:33551: Network error: Client connection negotiation failed: client connection to 127.22.1.124:33551: connect: Connection refused (error 111)
W20260809 05:27:07.180701 24658 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260809 05:27:07.181066 24658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260809 05:27:07.181177 24658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260809 05:27:07.181298 24658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260809 05:27:07.188676 24658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260809 05:27:07.188761 24658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260809 05:27:07.188846 24658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260809 05:27:07.188912 24658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260809 05:27:07.188995 24658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260809 05:27:07.199337 24658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.1.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.1.126:35743,127.22.1.125:36083,127.22.1.124:33551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/krb5kdc/kudu_127.22.1.124.keytab
--principal=kudu/127.22.1.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.1.124:33551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.1.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 09 Aug 2026 03:51:24 UTC on ccycloud.quasar-zwepce.root.comops.site
build id 4575769
ASAN enabled
I20260809 05:27:07.200743 24658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260809 05:27:07.202103 24658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260809 05:27:07.213325 24671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:07.213454 24668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260809 05:27:07.213301 24666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260809 05:27:07.214012 24658 server_base.cc:1034] running on GCE node
I20260809 05:27:07.214774 24658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260809 05:27:07.216100 24658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260809 05:27:07.217307 24658 hybrid_clock.cc:648] HybridClock initialized: now 1786253227217239 us; error 43 us; skew 500 ppm
I20260809 05:27:07.217644 24658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260809 05:27:07.220669 24658 init.cc:377] Logged in from keytab as kudu/127.22.1.124@KRBTEST.COM (short username kudu)
I20260809 05:27:07.222081 24658 webserver.cc:460] Webserver started at http://127.22.1.124:43839/ using document root <none> and password file <none>
I20260809 05:27:07.222728 24658 fs_manager.cc:359] Metadata directory not provided
I20260809 05:27:07.222829 24658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260809 05:27:07.223129 24658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260809 05:27:07.226038 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data/instance:
uuid: "fae6be39b1cf43a282b9a9a70aac705d"
format_stamp: "Formatted at 2026-08-09 05:27:07 on dist-test-slave-rmhg"
server_key: "31e73b1a2047a4b3e28baf185dc7df8d"
server_key_iv: "a1ec340d8983dd2502da5cbdec4a4b8e"
server_key_version: "encryptionkey@0"
I20260809 05:27:07.226794 24658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal/instance:
uuid: "fae6be39b1cf43a282b9a9a70aac705d"
format_stamp: "Formatted at 2026-08-09 05:27:07 on dist-test-slave-rmhg"
server_key: "31e73b1a2047a4b3e28baf185dc7df8d"
server_key_iv: "a1ec340d8983dd2502da5cbdec4a4b8e"
server_key_version: "encryptionkey@0"
I20260809 05:27:07.231734 24658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260809 05:27:07.235195 24682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260809 05:27:07.236591 24658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260809 05:27:07.236765 24658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data,/tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal
uuid: "fae6be39b1cf43a282b9a9a70aac705d"
format_stamp: "Formatted at 2026-08-09 05:27:07 on dist-test-slave-rmhg"
server_key: "31e73b1a2047a4b3e28baf185dc7df8d"
server_key_iv: "a1ec340d8983dd2502da5cbdec4a4b8e"
server_key_version: "encryptionkey@0"
I20260809 05:27:07.236974 24658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260809 05:27:07.263978 24658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260809 05:27:07.268189 24658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260809 05:27:07.268438 24658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260809 05:27:07.285759 24658 rpc_server.cc:307] RPC server started. Bound to: 127.22.1.124:33551
I20260809 05:27:07.285777 24753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.22.1.124:33551 every 8 connection(s)
I20260809 05:27:07.287091 24658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/data/info.pb
I20260809 05:27:07.290777 24755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260809 05:27:07.290905 22533 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu as pid 24658
I20260809 05:27:07.291095 22533 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGKhvF4/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786252853919692-22533-0/minicluster-data/master-2/wal/instance
I20260809 05:27:07.291518 22533 external_mini_cluster.cc:1428] Setting key 1bcd11300a6d8e99c8a1853277edf5a7
I20260809 05:27:07.300084 24755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
I20260809 05:27:07.315001 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.293253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.22.1.124:33551)
0809 05:27:07.294013 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.294024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.294068 (+ 44us) server_negotiation.cc:407] Connection header received
0809 05:27:07.295525 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.295548 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.295948 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.296278 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.298079 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.299462 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.301346 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.302013 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.302142 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.306998 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.307011 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.307029 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.307077 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.310359 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.311200 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.311207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.311215 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.311367 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.311782 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.311786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.311788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.312385 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.312711 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.313071 (+ 360us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.313521 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":338,"threads_started":1}
I20260809 05:27:07.318843 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.303123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.126:35743 (local address 127.0.0.1:53324)
0809 05:27:07.304030 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.304079 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:07.304263 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:07.304830 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:07.304838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:07.305654 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:07.305900 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.305910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.307035 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.307039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.308177 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.308193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.309142 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.309147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.309312 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.309875 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:07.309894 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:07.312204 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:07.315875 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.315885 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.315891 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.316467 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.316908 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.316911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.316914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.317055 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.317701 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:07.317709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:07.317937 (+ 228us) client_negotiation.cc:769] Sending connection context
0809 05:27:07.318130 (+ 193us) client_negotiation.cc:240] Negotiation successful
0809 05:27:07.318450 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":479,"threads_started":1}
I20260809 05:27:07.318904 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.303096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.22.1.126:35743)
0809 05:27:07.303617 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.303623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.304195 (+ 572us) server_negotiation.cc:407] Connection header received
0809 05:27:07.304514 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.304520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.304626 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.304724 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.306082 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.306863 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.308338 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.308935 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.309093 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.312405 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.312412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.312417 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.312515 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.315743 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.316611 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.316616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.316619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.316795 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.317183 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.317188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.317191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.317560 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.317715 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.318282 (+ 567us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.318495 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260809 05:27:07.320828 24755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.126@KRBTEST.COM
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:07.329743 24542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 4
I20260809 05:27:07.334975 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.318577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.22.1.126:35743)
0809 05:27:07.319167 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.319173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.319192 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:27:07.319250 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.319255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.319349 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.319454 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.321014 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.321899 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.323441 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.324048 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.324136 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.328591 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.328596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.328600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.328633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.331900 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.332760 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.332766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.332769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.332929 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.333382 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.333387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.333390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.333778 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.334100 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.334375 (+ 275us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.334608 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":894,"threads_started":1}
I20260809 05:27:07.334972 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.321572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.125:36083 (local address 127.0.0.1:35938)
0809 05:27:07.321722 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.321739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:07.321819 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:07.322117 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:07.322122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:07.322582 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:07.322764 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.322775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.323867 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.323871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.325050 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.325067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.325844 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.325851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.326055 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.326576 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:07.326598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:07.328656 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:07.332150 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.332155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.332158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.332670 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.333136 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.333141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.333145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.333353 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.334028 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:07.334032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:07.334195 (+ 163us) client_negotiation.cc:769] Sending connection context
0809 05:27:07.334328 (+ 133us) client_negotiation.cc:240] Negotiation successful
0809 05:27:07.334620 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260809 05:27:07.335868 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.321609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.22.1.125:36083)
0809 05:27:07.321792 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.321797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.321816 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:27:07.321901 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.321906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.321993 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.322079 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.322908 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.323715 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.325251 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.325684 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.325785 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.328839 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.328844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.328847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.328879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.331999 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.332807 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.332812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.332815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.332996 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.333491 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.333496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.333499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.333831 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.333961 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.335295 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.335533 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260809 05:27:07.336889 24755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } has no permanent_uuid. Determining permanent_uuid...
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.126@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:27:07.345520 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.330619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.22.1.124:33551)
0809 05:27:07.330809 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.330815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.330832 (+ 17us) server_negotiation.cc:407] Connection header received
0809 05:27:07.330879 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.330883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.330953 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.331049 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.331996 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.332768 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.334430 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.335223 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.335378 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.339445 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.339451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.339455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.339499 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.342875 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.343494 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.343500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.343503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.343637 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.344101 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.344107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.344110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.344401 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.344551 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.344918 (+ 367us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.345207 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260809 05:27:07.345556 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.330380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52586)
0809 05:27:07.330557 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.330580 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:07.330677 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:07.331113 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:07.331118 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:07.331616 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:07.331831 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.331844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.332910 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.332916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.334269 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.334285 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.335389 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.335395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.335598 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.336357 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:07.336379 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:07.339215 (+ 2836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:07.342999 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.343005 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.343009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.343377 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.343760 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.343765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.343769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.343955 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.344545 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:07.344550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:07.344684 (+ 134us) client_negotiation.cc:769] Sending connection context
0809 05:27:07.344803 (+ 119us) client_negotiation.cc:240] Negotiation successful
0809 05:27:07.345100 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253227, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.124@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.1.125@KRBTEST.COM
I20260809 05:27:07.351975 24542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: Bootstrap starting.
I20260809 05:27:07.353083 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.337566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52598)
0809 05:27:07.337706 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.337724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:07.337813 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:07.338675 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:07.338679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:07.339086 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:07.339215 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.339225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.340736 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.340742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.342590 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.342606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.343371 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.343376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.343569 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.344082 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:07.344104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:07.346788 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:07.350411 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.350417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.350421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.350916 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.351366 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.351370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.351374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.351537 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.352216 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:07.352220 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:07.352380 (+ 160us) client_negotiation.cc:769] Sending connection context
0809 05:27:07.352527 (+ 147us) client_negotiation.cc:240] Negotiation successful
0809 05:27:07.352811 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260809 05:27:07.353811 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.337866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.22.1.124:33551)
0809 05:27:07.338426 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.338430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.338451 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:27:07.338494 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.338497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.338546 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.338634 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.339368 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.340595 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.342735 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.343225 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.343331 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.347017 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.347025 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.347029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.347065 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.350272 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.351080 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.351084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.351087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.351247 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.351651 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.351656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.351659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.352022 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.352166 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.353372 (+ 1206us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.353551 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":288,"threads_started":1}
I20260809 05:27:07.355409 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.337391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.22.1.125:36083)
0809 05:27:07.337568 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.337573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.337591 (+ 18us) server_negotiation.cc:407] Connection header received
0809 05:27:07.337805 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.337810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.337886 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.337967 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.339802 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.340847 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.342189 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.342622 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.342733 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.348830 (+ 6097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.348836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.348841 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.348886 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.352186 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.353042 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.353047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.353051 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.353254 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.353709 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.353714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.353717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.354071 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.354347 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.354680 (+ 333us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.354923 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:27:07.356832 24542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:07.358197 24542 log.cc:824] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:07.359269 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: Bootstrap starting.
I20260809 05:27:07.361006 22533 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260809 05:27:07.361824 24542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: No bootstrap required, opened a new log
I20260809 05:27:07.362735 24755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:07.363833 24755 log.cc:824] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:07.364104 24503 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b4c618aa08604239a571be3b35a63cfa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53340
I20260809 05:27:07.366597 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: No bootstrap required, opened a new log
I20260809 05:27:07.371681 24542 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.372047 24542 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c618aa08604239a571be3b35a63cfa, State: Initialized, Role: FOLLOWER
I20260809 05:27:07.372536 24542 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.373456 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.373874 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.373914 24755 raft_consensus.cc:348] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.374248 24755 raft_consensus.cc:729] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae6be39b1cf43a282b9a9a70aac705d, State: Initialized, Role: FOLLOWER
I20260809 05:27:07.374687 24542 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:07.374730 24503 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260809 05:27:07.374656 24755 consensus_queue.cc:260] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.374902 24503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:07.375455 24773 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.375842 24773 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.376765 24755 sys_catalog.cc:564] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:07.378530 24503 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.380218 24503 leader_election.cc:288] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [CANDIDATE]: Term 1 election: Requested vote from peers 840635203bbb4cb88d979910e5bf4420 (127.22.1.125:36083), fae6be39b1cf43a282b9a9a70aac705d (127.22.1.124:33551)
I20260809 05:27:07.381678 24721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4c618aa08604239a571be3b35a63cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fae6be39b1cf43a282b9a9a70aac705d"
I20260809 05:27:07.382177 24721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:07.386711 24721 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c618aa08604239a571be3b35a63cfa in term 1.
I20260809 05:27:07.387706 24467 leader_election.cc:302] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4c618aa08604239a571be3b35a63cfa, fae6be39b1cf43a282b9a9a70aac705d; no voters:
I20260809 05:27:07.388274 24771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 1 FOLLOWER]: Leader election won for term 1
I20260809 05:27:07.389269 24771 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [term 1 LEADER]: Becoming Leader. State: Replica: b4c618aa08604239a571be3b35a63cfa, State: Running, Role: LEADER
I20260809 05:27:07.389803 24771 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
W20260809 05:27:07.399660 24792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260809 05:27:07.399787 24792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260809 05:27:07.400971 24798 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c618aa08604239a571be3b35a63cfa. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.401463 24798 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:07.398681 24445 master_runner.cc:186] Error getting master registration for 127.22.1.125:36083: OK, instance_id { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" instance_seqno: 1786253226736417 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:27:07.411350 24445 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 05:27:07.415433 24802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260809 05:27:07.417520 24802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260809 05:27:07.433300 24471 tablet.cc:2369] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260809 05:27:07.434577 24721 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [term 1 FOLLOWER]: Refusing update from remote peer b4c618aa08604239a571be3b35a63cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:27:07.435738 24798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:27:07.454116 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.420138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.22.1.126:35743)
0809 05:27:07.420320 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.420327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.426206 (+ 5879us) server_negotiation.cc:407] Connection header received
0809 05:27:07.426723 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.426730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.426826 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.426898 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.432717 (+ 5819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.433803 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.435829 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.436520 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.436623 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.445047 (+ 8424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.445054 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.445060 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.445109 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.448943 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.451434 (+ 2491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.451440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.451444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.451640 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.452130 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.452136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.452139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.452580 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.452838 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.453441 (+ 603us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.453722 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260809 05:27:07.467029 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.420138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.22.1.125:36083)
0809 05:27:07.420321 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.420328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.438067 (+ 17739us) server_negotiation.cc:407] Connection header received
0809 05:27:07.438710 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.438716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.438805 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.438879 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.441106 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.442298 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.455038 (+ 12740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.455748 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.455853 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.459770 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.459778 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.459783 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.459838 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.463397 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.464489 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.464495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.464499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.464678 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.465163 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.465168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.465171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.465582 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.465822 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.466396 (+ 574us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.466663 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260809 05:27:07.469051 24773 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c618aa08604239a571be3b35a63cfa. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.469413 24773 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.471732 24810 mvcc.cc:205] Tried to move back new op lower bound from 7316493219560390656 to 7316493219392786432. Current Snapshot: MvccSnapshot[applied={T|T < 7316493219560390656}]
I20260809 05:27:07.474086 24647 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } attempt: 5
Aug 09 05:27:07 dist-test-slave-rmhg krb5kdc[24423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786253226, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.1.125@KRBTEST.COM for kudu/127.22.1.124@KRBTEST.COM
I20260809 05:27:07.490940 24765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.438662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.22.1.124:33551)
0809 05:27:07.438943 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.438949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.450095 (+ 11146us) server_negotiation.cc:407] Connection header received
0809 05:27:07.450541 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.450548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.450682 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.450753 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.454341 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.455437 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.461426 (+ 5989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.462139 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.462253 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.468461 (+ 6208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.468468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.468473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.468515 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.473332 (+ 4817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.484389 (+ 11057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.484395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.484400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.484602 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.487710 (+ 3108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.487716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.487720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.488158 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.488396 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.490320 (+ 1924us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.490566 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:27:07.497812 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.474659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.1.124:33551 (local address 127.0.0.1:52618)
0809 05:27:07.475042 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0809 05:27:07.475062 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0809 05:27:07.475177 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0809 05:27:07.476290 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0809 05:27:07.476296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0809 05:27:07.476852 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0809 05:27:07.477089 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.477104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.478611 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.478616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.479960 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0809 05:27:07.479979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.481303 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.481309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0809 05:27:07.481552 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.482177 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0809 05:27:07.482200 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0809 05:27:07.485402 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0809 05:27:07.492969 (+ 7567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.492975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.492980 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.493589 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.494079 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0809 05:27:07.494085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0809 05:27:07.494089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0809 05:27:07.494261 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0809 05:27:07.496736 (+ 2475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0809 05:27:07.496742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0809 05:27:07.496923 (+ 181us) client_negotiation.cc:769] Sending connection context
0809 05:27:07.497069 (+ 146us) client_negotiation.cc:240] Negotiation successful
0809 05:27:07.497405 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260809 05:27:07.498799 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.475521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.22.1.124:33551)
0809 05:27:07.475734 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.475739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.475758 (+ 19us) server_negotiation.cc:407] Connection header received
0809 05:27:07.475812 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.475817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.475899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.476008 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.477279 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.478428 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.480146 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.480896 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.481010 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.485625 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0809 05:27:07.485633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0809 05:27:07.485638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0809 05:27:07.485675 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0809 05:27:07.492806 (+ 7131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.493778 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.493783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.493787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.493929 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0809 05:27:07.494390 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0809 05:27:07.494395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0809 05:27:07.494398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0809 05:27:07.496516 (+ 2118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0809 05:27:07.496669 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.498218 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.498451 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:27:07.505227 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: Bootstrap starting.
I20260809 05:27:07.510200 24647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: Neither blocks nor log segments found. Creating new log.
I20260809 05:27:07.511582 24647 log.cc:824] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: Log is configured to *not* fsync() on all Append() calls
I20260809 05:27:07.515811 24773 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.516124 24773 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.522296 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.523074 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:07.525271 24802 catalog_manager.cc:1347] Generated new cluster ID: 963877e7f23240338123563e8b8189d9
I20260809 05:27:07.525375 24802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260809 05:27:07.525490 24647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: No bootstrap required, opened a new log
I20260809 05:27:07.538278 24647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.538736 24647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840635203bbb4cb88d979910e5bf4420, State: Initialized, Role: FOLLOWER
I20260809 05:27:07.539283 24647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } }
I20260809 05:27:07.542774 24830 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260809 05:27:07.548516 24831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.553743 24647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: configured and running, proceeding with master startup.
I20260809 05:27:07.554294 24611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [term 0 FOLLOWER]: Advancing to term 1
I20260809 05:27:07.548982 24831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.557735 24612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b4c618aa08604239a571be3b35a63cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "840635203bbb4cb88d979910e5bf4420"
I20260809 05:27:07.549237 24658 master_runner.cc:186] Error getting master registration for 127.22.1.125:36083: OK, instance_id { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" instance_seqno: 1786253226736417 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260809 05:27:07.559753 24658 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260809 05:27:07.561321 24611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [term 1 FOLLOWER]: Refusing update from remote peer b4c618aa08604239a571be3b35a63cfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260809 05:27:07.564646 24612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c618aa08604239a571be3b35a63cfa in term 1.
I20260809 05:27:07.566774 24802 catalog_manager.cc:1370] Generated new certificate authority record
I20260809 05:27:07.566983 24798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260809 05:27:07.573961 24802 catalog_manager.cc:1504] Loading token signing keys...
I20260809 05:27:07.603545 24802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa: Generated new TSK 0
I20260809 05:27:07.604847 24802 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260809 05:27:07.622279 24831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4c618aa08604239a571be3b35a63cfa. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.622605 24831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.623373 24844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.623761 24844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420 [sys.catalog]: This master's current role is: FOLLOWER
I20260809 05:27:07.625029 24771 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4c618aa08604239a571be3b35a63cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4c618aa08604239a571be3b35a63cfa" member_type: VOTER last_known_addr { host: "127.22.1.126" port: 35743 } } peers { permanent_uuid: "840635203bbb4cb88d979910e5bf4420" member_type: VOTER last_known_addr { host: "127.22.1.125" port: 36083 } } peers { permanent_uuid: "fae6be39b1cf43a282b9a9a70aac705d" member_type: VOTER last_known_addr { host: "127.22.1.124" port: 33551 } } }
I20260809 05:27:07.625349 24771 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4c618aa08604239a571be3b35a63cfa [sys.catalog]: This master's current role is: LEADER
I20260809 05:27:07.664359 24862 catalog_manager.cc:1259] Loaded cluster ID: 963877e7f23240338123563e8b8189d9
I20260809 05:27:07.664522 24862 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: loading cluster ID for follower catalog manager: success
I20260809 05:27:07.668738 24862 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: acquiring CA information for follower catalog manager: success
I20260809 05:27:07.671582 24862 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 840635203bbb4cb88d979910e5bf4420: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260809 05:27:07.722365 24761 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.708231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.22.1.126:35743)
0809 05:27:07.708440 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.708445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.719053 (+ 10608us) server_negotiation.cc:407] Connection header received
0809 05:27:07.720586 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.720598 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.720690 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.720775 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.722107 (+ 1332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53350: BlockingRecv error: recv got EOF from 127.0.0.1:53350 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 05:27:07.726541 24653 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.707515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.22.1.125:36083)
0809 05:27:07.707963 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.707969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.724268 (+ 16299us) server_negotiation.cc:407] Connection header received
0809 05:27:07.725038 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.725044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.725172 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.725251 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.726296 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35964: BlockingRecv error: recv got EOF from 127.0.0.1:35964 (error 108)
Metrics: {"server-negotiator.queue_time_us":247}
W20260809 05:27:07.730032 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.719758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.22.1.124:33551)
0809 05:27:07.719937 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.719942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.727565 (+ 7623us) server_negotiation.cc:407] Connection header received
0809 05:27:07.728193 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.728199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.728286 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.728357 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.729785 (+ 1428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52634: BlockingRecv error: recv got EOF from 127.0.0.1:52634 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:27:07.743804 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.735092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.22.1.125:36083)
0809 05:27:07.735267 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.735273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.735779 (+ 506us) server_negotiation.cc:407] Connection header received
0809 05:27:07.736436 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.736442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.736663 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.736868 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.737306 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.738473 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.740570 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.741364 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.741504 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.742273 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.742948 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.743084 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.743348 (+ 264us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.743454 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260809 05:27:07.781651 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.779332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.22.1.124:33551)
0809 05:27:07.779529 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.779536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.780032 (+ 496us) server_negotiation.cc:407] Connection header received
0809 05:27:07.780288 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.780292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.780397 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.780497 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.781362 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52640: BlockingRecv error: recv got EOF from 127.0.0.1:52640 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260809 05:27:07.791390 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.786068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.22.1.125:36083)
0809 05:27:07.786265 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.786271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.786780 (+ 509us) server_negotiation.cc:407] Connection header received
0809 05:27:07.786979 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.786983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.787077 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.787170 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.787599 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.788373 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.789729 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.790176 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.790287 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.790667 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.790788 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.790865 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.790999 (+ 134us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.791079 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260809 05:27:07.803135 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.797842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.22.1.124:33551)
0809 05:27:07.798055 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.798063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.801728 (+ 3665us) server_negotiation.cc:407] Connection header received
0809 05:27:07.801932 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.801937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.802012 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.802099 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.802874 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52650: BlockingRecv error: recv got EOF from 127.0.0.1:52650 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260809 05:27:07.813078 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.807628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.22.1.125:36083)
0809 05:27:07.807834 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.807841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.808328 (+ 487us) server_negotiation.cc:407] Connection header received
0809 05:27:07.808520 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.808526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.808603 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.808678 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.809202 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.810039 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.811393 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.811879 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.811996 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.812361 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.812471 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.812551 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.812679 (+ 128us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.812757 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260809 05:27:07.820884 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.818670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.22.1.124:33551)
0809 05:27:07.818865 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.818872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.819367 (+ 495us) server_negotiation.cc:407] Connection header received
0809 05:27:07.819554 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.819559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.819655 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.819720 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.820658 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52664: BlockingRecv error: recv got EOF from 127.0.0.1:52664 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260809 05:27:07.833843 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.828448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.22.1.125:36083)
0809 05:27:07.828724 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.828731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.829260 (+ 529us) server_negotiation.cc:407] Connection header received
0809 05:27:07.829465 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.829470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.829538 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.829625 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.830060 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.830819 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.832151 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.832615 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.832723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.833107 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.833248 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.833325 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.833466 (+ 141us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.833550 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
W20260809 05:27:07.843088 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.839862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.22.1.124:33551)
0809 05:27:07.840156 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.840164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.841709 (+ 1545us) server_negotiation.cc:407] Connection header received
0809 05:27:07.841913 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.841920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.842009 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.842112 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.842868 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52680: BlockingRecv error: recv got EOF from 127.0.0.1:52680 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260809 05:27:07.864573 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.859082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.22.1.125:36083)
0809 05:27:07.859292 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.859299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.859822 (+ 523us) server_negotiation.cc:407] Connection header received
0809 05:27:07.860014 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.860020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.860124 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.860215 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.860653 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.861462 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.862824 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.863275 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.863384 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.863786 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.863929 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.864006 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.864160 (+ 154us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.864246 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260809 05:27:07.872742 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.870979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.22.1.124:33551)
0809 05:27:07.871185 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.871192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.871331 (+ 139us) server_negotiation.cc:407] Connection header received
0809 05:27:07.871605 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.871610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.871683 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.871767 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.872488 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52690: BlockingRecv error: recv got EOF from 127.0.0.1:52690 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260809 05:27:07.896996 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.891634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.22.1.125:36083)
0809 05:27:07.891855 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.891862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.892371 (+ 509us) server_negotiation.cc:407] Connection header received
0809 05:27:07.892571 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.892575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.892649 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.892727 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.893177 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.893975 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.895277 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.895727 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.895834 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.896255 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.896401 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.896480 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.896625 (+ 145us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.896717 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260809 05:27:07.904915 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.902835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.22.1.124:33551)
0809 05:27:07.903046 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.903053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.903525 (+ 472us) server_negotiation.cc:407] Connection header received
0809 05:27:07.903697 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.903703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.903782 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.903865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.904691 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52692: BlockingRecv error: recv got EOF from 127.0.0.1:52692 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260809 05:27:07.945864 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:07.940217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.22.1.125:36083)
0809 05:27:07.940471 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.940479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.941041 (+ 562us) server_negotiation.cc:407] Connection header received
0809 05:27:07.941305 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.941311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.941398 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.941486 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:07.941991 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.942742 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.944104 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:07.944576 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:07.944700 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:07.945081 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:07.945227 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:07.945310 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:07.945453 (+ 143us) server_negotiation.cc:299] Negotiation successful
0809 05:27:07.945532 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260809 05:27:07.953464 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:07.951567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.22.1.124:33551)
0809 05:27:07.951745 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:07.951750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:07.951823 (+ 73us) server_negotiation.cc:407] Connection header received
0809 05:27:07.952003 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:07.952006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:07.952077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:07.952168 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:07.953192 (+ 1024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52706: BlockingRecv error: recv got EOF from 127.0.0.1:52706 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260809 05:27:08.027002 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.021058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.22.1.125:36083)
0809 05:27:08.021299 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.021307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.021585 (+ 278us) server_negotiation.cc:407] Connection header received
0809 05:27:08.021760 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.021770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.021867 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.022007 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.022469 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.023660 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.025009 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.025718 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.025858 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:08.026241 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:08.026395 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:08.026503 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:08.026592 (+ 89us) server_negotiation.cc:299] Negotiation successful
0809 05:27:08.026679 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260809 05:27:08.040907 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:08.038675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.22.1.124:33551)
0809 05:27:08.038922 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.038928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.039332 (+ 404us) server_negotiation.cc:407] Connection header received
0809 05:27:08.039556 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.039560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.039638 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.039723 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:08.040657 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52714: BlockingRecv error: recv got EOF from 127.0.0.1:52714 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260809 05:27:08.177726 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.172676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.22.1.125:36083)
0809 05:27:08.172844 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.172849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.173193 (+ 344us) server_negotiation.cc:407] Connection header received
0809 05:27:08.173398 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.173402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.173476 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.173572 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.173950 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.174663 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.176068 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.176555 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.176650 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:08.177030 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:08.177217 (+ 187us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:08.177286 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:08.177416 (+ 130us) server_negotiation.cc:299] Negotiation successful
0809 05:27:08.177498 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260809 05:27:08.184952 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:08.183190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.22.1.124:33551)
0809 05:27:08.183350 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.183355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.183670 (+ 315us) server_negotiation.cc:407] Connection header received
0809 05:27:08.183822 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.183826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.183899 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.183986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:08.184708 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52722: BlockingRecv error: recv got EOF from 127.0.0.1:52722 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260809 05:27:08.450524 24653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.445350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.22.1.125:36083)
0809 05:27:08.445605 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.445612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.445789 (+ 177us) server_negotiation.cc:407] Connection header received
0809 05:27:08.446024 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.446029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.446101 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.446183 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.446674 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.447396 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.448759 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.449204 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.449318 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:08.449785 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:08.449903 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:08.449991 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:08.450161 (+ 170us) server_negotiation.cc:299] Negotiation successful
0809 05:27:08.450248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260809 05:27:08.458138 24760 negotiation.cc:337] Failed RPC negotiation. Trace:
0809 05:27:08.456375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52726 (local address 127.22.1.124:33551)
0809 05:27:08.456601 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.456607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.456732 (+ 125us) server_negotiation.cc:407] Connection header received
0809 05:27:08.456937 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.456941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.457018 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.457106 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0809 05:27:08.457939 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52726: BlockingRecv error: recv got EOF from 127.0.0.1:52726 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260809 05:27:08.547267 24830 catalog_manager.cc:1259] Loaded cluster ID: 963877e7f23240338123563e8b8189d9
I20260809 05:27:08.547389 24830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: loading cluster ID for follower catalog manager: success
I20260809 05:27:08.550158 24830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: acquiring CA information for follower catalog manager: success
I20260809 05:27:08.552062 24830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fae6be39b1cf43a282b9a9a70aac705d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260809 05:27:08.978850 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.973725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.22.1.125:36083)
0809 05:27:08.974195 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.974199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.974285 (+ 86us) server_negotiation.cc:407] Connection header received
0809 05:27:08.974463 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.974468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.974547 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.974647 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.975055 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.975845 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.977226 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.977672 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.977773 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:08.978178 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:08.978299 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:08.978366 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:08.978472 (+ 106us) server_negotiation.cc:299] Negotiation successful
0809 05:27:08.978546 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260809 05:27:08.989660 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.984300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.22.1.124:33551)
0809 05:27:08.984711 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.984716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.984814 (+ 98us) server_negotiation.cc:407] Connection header received
0809 05:27:08.984966 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.984976 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.985141 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.985284 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.985640 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.986322 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.987605 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.988015 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:08.988107 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:08.988521 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:08.989105 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:08.989273 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:08.989347 (+ 74us) server_negotiation.cc:299] Negotiation successful
0809 05:27:08.989416 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":211,"threads_started":1}
I20260809 05:27:09.002666 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.995643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.22.1.125:36083)
0809 05:27:08.995890 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.995897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.997014 (+ 1117us) server_negotiation.cc:407] Connection header received
0809 05:27:08.997225 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.997231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.997347 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.997430 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.997995 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.999037 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.000777 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:09.001221 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.001326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:09.001787 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:09.001931 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:09.002017 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:09.002248 (+ 231us) server_negotiation.cc:299] Negotiation successful
0809 05:27:09.002341 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260809 05:27:09.004650 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.995289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.22.1.126:35743)
0809 05:27:08.997581 (+ 2292us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.997588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.997609 (+ 21us) server_negotiation.cc:407] Connection header received
0809 05:27:08.997758 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.997763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.997988 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.998196 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.998892 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:08.999984 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.002116 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:09.002775 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.002948 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:09.003638 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:09.004018 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:09.004136 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:09.004238 (+ 102us) server_negotiation.cc:299] Negotiation successful
0809 05:27:09.004324 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2154,"thread_start_us":2031,"threads_started":1}
I20260809 05:27:09.004890 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0809 05:27:08.996038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.22.1.124:33551)
0809 05:27:08.996251 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0809 05:27:08.996258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0809 05:27:08.998944 (+ 2686us) server_negotiation.cc:407] Connection header received
0809 05:27:08.999112 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0809 05:27:08.999118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0809 05:27:08.999206 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0809 05:27:08.999281 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0809 05:27:08.999857 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:09.000897 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.002966 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0809 05:27:09.003506 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0809 05:27:09.003593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0809 05:27:09.004125 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0809 05:27:09.004273 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0809 05:27:09.004349 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0809 05:27:09.004572 (+ 223us) server_negotiation.cc:299] Negotiation successful
0809 05:27:09.004655 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260809 05:27:09.009195 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24445
I20260809 05:27:09.010193 24536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:09.119715 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24445
I20260809 05:27:09.148610 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24550
I20260809 05:27:09.149760 24641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:09.256593 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24550
I20260809 05:27:09.284781 22533 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu pid 24658
I20260809 05:27:09.285944 24749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260809 05:27:09.391712 22533 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGKhvF4/build/asan/bin/kudu with pid 24658
2026-08-09T05:27:09Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5181 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10048 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375363 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST